[==========] Running 7 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from LeaderRebalancerTest
[ RUN ] LeaderRebalancerTest.FunctionalTestForDivided
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260805 22:22:35.694414 29783 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.21.254:40149
I20260805 22:22:35.697450 29783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260805 22:22:35.699411 29783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:22:35.757332 29790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:22:35.758961 29791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:22:35.759186 29796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:22:37.074229 29793 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260805 22:22:37.074329 29783 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260805 22:22:37.078488 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260805 22:22:37.078758 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260805 22:22:37.078922 29783 hybrid_clock.cc:648] HybridClock initialized: now 1785968557078906 us; error 0 us; skew 500 ppm
I20260805 22:22:37.080331 29783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:22:37.094774 29783 webserver.cc:460] Webserver started at http://127.29.21.254:37999/ using document root <none> and password file <none>
I20260805 22:22:37.096341 29783 fs_manager.cc:359] Metadata directory not provided
I20260805 22:22:37.096589 29783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:22:37.097437 29783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:22:37.106427 29783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/master-0-root/instance:
uuid: "7dba7f8683554be786e28403f0117dfb"
format_stamp: "Formatted at 2026-08-05 22:22:37 on dist-test-slave-f0tn"
I20260805 22:22:37.119560 29783 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.006s sys 0.005s
I20260805 22:22:37.128368 29811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:22:37.130509 29783 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.000s
I20260805 22:22:37.130995 29783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/master-0-root
uuid: "7dba7f8683554be786e28403f0117dfb"
format_stamp: "Formatted at 2026-08-05 22:22:37 on dist-test-slave-f0tn"
I20260805 22:22:37.131392 29783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:22:37.177451 29783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:22:37.179530 29783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260805 22:22:37.180351 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:22:37.379339 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.254:40149
I20260805 22:22:37.379758 29898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.21.254:40149 every 8 connection(s)
I20260805 22:22:37.396381 29899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:37.449254 29899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb: Bootstrap starting.
I20260805 22:22:37.464337 29899 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:37.469027 29899 log.cc:824] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb: Log is configured to *not* fsync() on all Append() calls
I20260805 22:22:37.478668 29899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb: No bootstrap required, opened a new log
I20260805 22:22:37.520721 29899 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dba7f8683554be786e28403f0117dfb" member_type: VOTER }
I20260805 22:22:37.521545 29899 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:37.521981 29899 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dba7f8683554be786e28403f0117dfb, State: Initialized, Role: FOLLOWER
I20260805 22:22:37.523676 29899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dba7f8683554be786e28403f0117dfb" member_type: VOTER }
I20260805 22:22:37.524516 29899 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:22:37.524842 29899 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:22:37.525337 29899 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:37.531117 29899 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dba7f8683554be786e28403f0117dfb" member_type: VOTER }
I20260805 22:22:37.532555 29899 leader_election.cc:302] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dba7f8683554be786e28403f0117dfb; no voters:
I20260805 22:22:37.537663 29899 leader_election.cc:288] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:22:37.538113 29904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:22:37.543947 29904 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [term 1 LEADER]: Becoming Leader. State: Replica: 7dba7f8683554be786e28403f0117dfb, State: Running, Role: LEADER
I20260805 22:22:37.545058 29904 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dba7f8683554be786e28403f0117dfb" member_type: VOTER }
I20260805 22:22:37.546540 29899 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:22:37.562420 29906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dba7f8683554be786e28403f0117dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dba7f8683554be786e28403f0117dfb" member_type: VOTER } }
I20260805 22:22:37.562420 29907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dba7f8683554be786e28403f0117dfb. Latest consensus state: current_term: 1 leader_uuid: "7dba7f8683554be786e28403f0117dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dba7f8683554be786e28403f0117dfb" member_type: VOTER } }
I20260805 22:22:37.566428 29906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [sys.catalog]: This master's current role is: LEADER
I20260805 22:22:37.566323 29907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [sys.catalog]: This master's current role is: LEADER
I20260805 22:22:37.579232 29915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:22:37.602731 29915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:22:37.611632 29783 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260805 22:22:37.632380 29915 catalog_manager.cc:1347] Generated new cluster ID: eca067458b2d4ec9bb9c76576f8218ad
I20260805 22:22:37.632782 29915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:22:37.654440 29915 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:22:37.657553 29915 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:22:37.677234 29915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb: Generated new TSK 0
I20260805 22:22:37.678613 29915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:22:37.743340 29783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:22:37.757805 29929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:22:37.761448 29930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:22:37.771030 29935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:22:37.773247 29783 server_base.cc:1034] running on GCE node
I20260805 22:22:37.774343 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260805 22:22:37.774730 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260805 22:22:37.774986 29783 hybrid_clock.cc:648] HybridClock initialized: now 1785968557774968 us; error 0 us; skew 500 ppm
I20260805 22:22:37.775856 29783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:22:37.782809 29783 webserver.cc:460] Webserver started at http://127.29.21.193:45271/ using document root <none> and password file <none>
I20260805 22:22:37.783672 29783 fs_manager.cc:359] Metadata directory not provided
I20260805 22:22:37.783943 29783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:22:37.784313 29783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:22:37.786289 29783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/ts-0-root/instance:
uuid: "607da182484e4982a5d88b345ded418c"
format_stamp: "Formatted at 2026-08-05 22:22:37 on dist-test-slave-f0tn"
I20260805 22:22:37.795621 29783 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20260805 22:22:37.804845 29942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:22:37.805986 29783 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.004s
I20260805 22:22:37.806504 29783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/ts-0-root
uuid: "607da182484e4982a5d88b345ded418c"
format_stamp: "Formatted at 2026-08-05 22:22:37 on dist-test-slave-f0tn"
I20260805 22:22:37.807051 29783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:22:37.854144 29783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:22:37.856834 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:22:37.863034 29783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:22:37.872126 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:22:37.872550 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260805 22:22:37.872925 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:22:37.873173 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:22:38.055346 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.193:39793
I20260805 22:22:38.055473 30035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.21.193:39793 every 8 connection(s)
I20260805 22:22:38.079945 29783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:22:38.099542 30040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:22:38.104888 30045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:22:38.114691 30036 heartbeater.cc:347] Connected to a master server at 127.29.21.254:40149
I20260805 22:22:38.115302 30036 heartbeater.cc:464] Registering TS with master...
I20260805 22:22:38.116778 30036 heartbeater.cc:511] Master 127.29.21.254:40149 requested a full tablet report, sending...
W20260805 22:22:38.120821 30047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:22:38.122277 29783 server_base.cc:1034] running on GCE node
I20260805 22:22:38.122534 29837 ts_manager.cc:194] Registered new tserver with Master: 607da182484e4982a5d88b345ded418c (127.29.21.193:39793)
I20260805 22:22:38.123595 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260805 22:22:38.123898 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260805 22:22:38.124137 29783 hybrid_clock.cc:648] HybridClock initialized: now 1785968558124116 us; error 0 us; skew 500 ppm
I20260805 22:22:38.124939 29783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:22:38.126245 29837 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58470
I20260805 22:22:38.128509 29783 webserver.cc:460] Webserver started at http://127.29.21.194:40033/ using document root <none> and password file <none>
I20260805 22:22:38.129271 29783 fs_manager.cc:359] Metadata directory not provided
I20260805 22:22:38.129592 29783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:22:38.129874 29783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:22:38.131013 29783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/ts-1-root/instance:
uuid: "1daa858569c149c5a44f92fe8c9bc635"
format_stamp: "Formatted at 2026-08-05 22:22:38 on dist-test-slave-f0tn"
I20260805 22:22:38.136831 29783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260805 22:22:38.141431 30058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:22:38.142313 29783 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260805 22:22:38.142582 29783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/ts-1-root
uuid: "1daa858569c149c5a44f92fe8c9bc635"
format_stamp: "Formatted at 2026-08-05 22:22:38 on dist-test-slave-f0tn"
I20260805 22:22:38.142933 29783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:22:38.171672 29783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:22:38.173218 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:22:38.175347 29783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:22:38.178025 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:22:38.178236 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:22:38.178469 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:22:38.178601 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:22:38.416497 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.194:39069
I20260805 22:22:38.416651 30152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.21.194:39069 every 8 connection(s)
I20260805 22:22:38.423960 29783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:22:38.435949 30156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:22:38.439419 30157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:22:38.440054 30153 heartbeater.cc:347] Connected to a master server at 127.29.21.254:40149
I20260805 22:22:38.440541 30153 heartbeater.cc:464] Registering TS with master...
I20260805 22:22:38.441507 30153 heartbeater.cc:511] Master 127.29.21.254:40149 requested a full tablet report, sending...
I20260805 22:22:38.444020 29837 ts_manager.cc:194] Registered new tserver with Master: 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069)
I20260805 22:22:38.446255 29837 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58476
I20260805 22:22:38.450330 29783 server_base.cc:1034] running on GCE node
W20260805 22:22:38.452028 30160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:22:38.453621 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260805 22:22:38.453859 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260805 22:22:38.454015 29783 hybrid_clock.cc:648] HybridClock initialized: now 1785968558454002 us; error 0 us; skew 500 ppm
I20260805 22:22:38.454497 29783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:22:38.457278 29783 webserver.cc:460] Webserver started at http://127.29.21.195:36755/ using document root <none> and password file <none>
I20260805 22:22:38.457800 29783 fs_manager.cc:359] Metadata directory not provided
I20260805 22:22:38.457988 29783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:22:38.458242 29783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:22:38.459267 29783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/ts-2-root/instance:
uuid: "09c9abba79e64e92b71def50c1afbe27"
format_stamp: "Formatted at 2026-08-05 22:22:38 on dist-test-slave-f0tn"
I20260805 22:22:38.463842 29783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 22:22:38.467556 30165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:22:38.468374 29783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260805 22:22:38.468639 29783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/ts-2-root
uuid: "09c9abba79e64e92b71def50c1afbe27"
format_stamp: "Formatted at 2026-08-05 22:22:38 on dist-test-slave-f0tn"
I20260805 22:22:38.468889 29783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForDivided.1785968555326349-29783-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:22:38.485839 29783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:22:38.487107 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:22:38.489521 29783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:22:38.492893 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:22:38.493080 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:22:38.493317 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:22:38.493495 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260805 22:22:38.599663 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.195:39287
I20260805 22:22:38.599876 30247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.21.195:39287 every 8 connection(s)
I20260805 22:22:38.620870 30249 heartbeater.cc:347] Connected to a master server at 127.29.21.254:40149
I20260805 22:22:38.621266 30249 heartbeater.cc:464] Registering TS with master...
I20260805 22:22:38.622047 30249 heartbeater.cc:511] Master 127.29.21.254:40149 requested a full tablet report, sending...
I20260805 22:22:38.623973 29837 ts_manager.cc:194] Registered new tserver with Master: 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:38.624414 29783 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015828098s
I20260805 22:22:38.626024 29837 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:58488
I20260805 22:22:38.627666 29783 test_util.cc:268] Using random seed: 897866553
I20260805 22:22:38.686414 29837 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58500:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q""\004\001\000\216\3438\016\004\001\000\034\307q\034\004\001\000\252\252\252*\004\001\0008\216\3438\004\001\000\306q\034G\004\001\000TUUU\004\001\000\3428\216c\004\001\000p\034\307q"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:22:38.691449 29837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:22:38.808347 30104 tablet_service.cc:1467] Processing CreateTablet for tablet 777755da0c344319aaa4f32ec55ce02a (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260805 22:22:38.817520 30103 tablet_service.cc:1467] Processing CreateTablet for tablet c9ff840de5194b14935947f0ee7ee529 (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260805 22:22:38.812987 30102 tablet_service.cc:1467] Processing CreateTablet for tablet b676fd9c96e94b01b7e1a9b297d8221f (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260805 22:22:38.820809 30096 tablet_service.cc:1467] Processing CreateTablet for tablet 05d7ac208d874bdab1e7ee5d1909814d (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260805 22:22:38.824256 30201 tablet_service.cc:1467] Processing CreateTablet for tablet 777755da0c344319aaa4f32ec55ce02a (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260805 22:22:38.822052 30095 tablet_service.cc:1467] Processing CreateTablet for tablet 3e86514dd0434e69ae26b33066fc8bc0 (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260805 22:22:38.823385 30094 tablet_service.cc:1467] Processing CreateTablet for tablet a8f348c46b854206beab7393137dccb5 (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260805 22:22:38.827762 30104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 777755da0c344319aaa4f32ec55ce02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.822115 30101 tablet_service.cc:1467] Processing CreateTablet for tablet 2230cd3dbabd47788d44be34d3d596c2 (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260805 22:22:38.830331 30200 tablet_service.cc:1467] Processing CreateTablet for tablet c9ff840de5194b14935947f0ee7ee529 (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260805 22:22:38.828576 30093 tablet_service.cc:1467] Processing CreateTablet for tablet a7d64cdb99414419b519f3abb289edb1 (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260805 22:22:38.832636 30201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 777755da0c344319aaa4f32ec55ce02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.833289 30199 tablet_service.cc:1467] Processing CreateTablet for tablet b676fd9c96e94b01b7e1a9b297d8221f (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260805 22:22:38.828065 30097 tablet_service.cc:1467] Processing CreateTablet for tablet 6267e585b62945d492c1fbb3c737c38d (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260805 22:22:38.835541 30096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05d7ac208d874bdab1e7ee5d1909814d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.840036 30198 tablet_service.cc:1467] Processing CreateTablet for tablet 2230cd3dbabd47788d44be34d3d596c2 (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260805 22:22:38.844377 30200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9ff840de5194b14935947f0ee7ee529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.843796 30197 tablet_service.cc:1467] Processing CreateTablet for tablet 6267e585b62945d492c1fbb3c737c38d (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260805 22:22:38.845546 30095 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e86514dd0434e69ae26b33066fc8bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.847399 30101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2230cd3dbabd47788d44be34d3d596c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.848974 30198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2230cd3dbabd47788d44be34d3d596c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.849906 30097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6267e585b62945d492c1fbb3c737c38d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.849174 30193 tablet_service.cc:1467] Processing CreateTablet for tablet 3e86514dd0434e69ae26b33066fc8bc0 (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260805 22:22:38.849215 30195 tablet_service.cc:1467] Processing CreateTablet for tablet 05d7ac208d874bdab1e7ee5d1909814d (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260805 22:22:38.860414 30102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b676fd9c96e94b01b7e1a9b297d8221f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.862502 30193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e86514dd0434e69ae26b33066fc8bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.856056 29985 tablet_service.cc:1467] Processing CreateTablet for tablet 777755da0c344319aaa4f32ec55ce02a (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION VALUES < 238609294
I20260805 22:22:38.867017 29981 tablet_service.cc:1467] Processing CreateTablet for tablet 2230cd3dbabd47788d44be34d3d596c2 (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 715827882 <= VALUES < 954437176
I20260805 22:22:38.862684 29983 tablet_service.cc:1467] Processing CreateTablet for tablet b676fd9c96e94b01b7e1a9b297d8221f (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 477218588 <= VALUES < 715827882
I20260805 22:22:38.861815 30192 tablet_service.cc:1467] Processing CreateTablet for tablet a8f348c46b854206beab7393137dccb5 (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260805 22:22:38.870020 30197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6267e585b62945d492c1fbb3c737c38d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.874234 30094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f348c46b854206beab7393137dccb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.873278 30191 tablet_service.cc:1467] Processing CreateTablet for tablet a7d64cdb99414419b519f3abb289edb1 (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260805 22:22:38.861848 29984 tablet_service.cc:1467] Processing CreateTablet for tablet c9ff840de5194b14935947f0ee7ee529 (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 238609294 <= VALUES < 477218588
I20260805 22:22:38.878569 29981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2230cd3dbabd47788d44be34d3d596c2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.880460 29978 tablet_service.cc:1467] Processing CreateTablet for tablet 6267e585b62945d492c1fbb3c737c38d (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 954437176 <= VALUES < 1193046470
I20260805 22:22:38.885207 29985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 777755da0c344319aaa4f32ec55ce02a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.886065 29975 tablet_service.cc:1467] Processing CreateTablet for tablet 05d7ac208d874bdab1e7ee5d1909814d (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 1193046470 <= VALUES < 1431655764
I20260805 22:22:38.890188 29974 tablet_service.cc:1467] Processing CreateTablet for tablet 3e86514dd0434e69ae26b33066fc8bc0 (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 1431655764 <= VALUES < 1670265058
I20260805 22:22:38.888528 30093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d64cdb99414419b519f3abb289edb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.896152 30199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b676fd9c96e94b01b7e1a9b297d8221f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.897393 29978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6267e585b62945d492c1fbb3c737c38d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.908754 30191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d64cdb99414419b519f3abb289edb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.912443 29972 tablet_service.cc:1467] Processing CreateTablet for tablet a7d64cdb99414419b519f3abb289edb1 (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 1908874352 <= VALUES
I20260805 22:22:38.914561 29983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b676fd9c96e94b01b7e1a9b297d8221f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.917008 30103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9ff840de5194b14935947f0ee7ee529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.913494 29973 tablet_service.cc:1467] Processing CreateTablet for tablet a8f348c46b854206beab7393137dccb5 (DEFAULT_TABLE table=test-workload [id=43648f84eaae455ab2112de82622458a]), partition=RANGE (key) PARTITION 1670265058 <= VALUES < 1908874352
I20260805 22:22:38.924993 29975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05d7ac208d874bdab1e7ee5d1909814d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.923303 30195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05d7ac208d874bdab1e7ee5d1909814d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.930368 29972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7d64cdb99414419b519f3abb289edb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.938090 30192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f348c46b854206beab7393137dccb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.944847 29973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a8f348c46b854206beab7393137dccb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:38.972019 29984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9ff840de5194b14935947f0ee7ee529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:39.032397 30275 tablet_bootstrap.cc:492] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635: Bootstrap starting.
I20260805 22:22:39.127732 30277 tablet_bootstrap.cc:492] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c: Bootstrap starting.
I20260805 22:22:39.134460 30036 heartbeater.cc:503] Master 127.29.21.254:40149 was elected leader, sending a full tablet report...
I20260805 22:22:39.151844 30275 tablet_bootstrap.cc:654] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.152056 30277 tablet_bootstrap.cc:654] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.153880 30278 tablet_bootstrap.cc:492] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27: Bootstrap starting.
I20260805 22:22:39.157495 29974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e86514dd0434e69ae26b33066fc8bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:22:39.171779 30277 tablet_bootstrap.cc:492] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c: No bootstrap required, opened a new log
I20260805 22:22:39.172305 30277 ts_tablet_manager.cc:1397] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c: Time spent bootstrapping tablet: real 0.085s user 0.009s sys 0.010s
I20260805 22:22:39.175164 30277 raft_consensus.cc:348] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.176011 30277 raft_consensus.cc:374] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.176337 30277 raft_consensus.cc:729] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607da182484e4982a5d88b345ded418c, State: Initialized, Role: FOLLOWER
I20260805 22:22:39.177067 30277 consensus_queue.cc:260] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.187956 30277 ts_tablet_manager.cc:1428] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
I20260805 22:22:39.189052 30277 tablet_bootstrap.cc:492] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c: Bootstrap starting.
I20260805 22:22:39.196223 30277 tablet_bootstrap.cc:654] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.200186 30278 tablet_bootstrap.cc:654] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.211172 30275 tablet_bootstrap.cc:492] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635: No bootstrap required, opened a new log
I20260805 22:22:39.211658 30275 ts_tablet_manager.cc:1397] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635: Time spent bootstrapping tablet: real 0.189s user 0.013s sys 0.050s
I20260805 22:22:39.214973 30278 tablet_bootstrap.cc:492] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27: No bootstrap required, opened a new log
I20260805 22:22:39.215083 30275 raft_consensus.cc:348] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:39.215569 30278 ts_tablet_manager.cc:1397] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27: Time spent bootstrapping tablet: real 0.067s user 0.011s sys 0.004s
I20260805 22:22:39.215850 30275 raft_consensus.cc:374] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.216161 30275 raft_consensus.cc:729] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Initialized, Role: FOLLOWER
I20260805 22:22:39.216849 30275 consensus_queue.cc:260] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:39.218842 30278 raft_consensus.cc:348] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:39.219727 30278 raft_consensus.cc:374] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.220075 30278 raft_consensus.cc:729] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Initialized, Role: FOLLOWER
I20260805 22:22:39.220268 30277 tablet_bootstrap.cc:492] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c: No bootstrap required, opened a new log
I20260805 22:22:39.220750 30277 ts_tablet_manager.cc:1397] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.024s
I20260805 22:22:39.221122 30278 consensus_queue.cc:260] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:39.223840 30277 raft_consensus.cc:348] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.224601 30277 raft_consensus.cc:374] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.224891 30277 raft_consensus.cc:729] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607da182484e4982a5d88b345ded418c, State: Initialized, Role: FOLLOWER
I20260805 22:22:39.225623 30277 consensus_queue.cc:260] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.228165 30277 ts_tablet_manager.cc:1428] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c: Time spent starting tablet: real 0.007s user 0.004s sys 0.004s
I20260805 22:22:39.229154 30277 tablet_bootstrap.cc:492] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c: Bootstrap starting.
I20260805 22:22:39.235940 30277 tablet_bootstrap.cc:654] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.238651 30153 heartbeater.cc:503] Master 127.29.21.254:40149 was elected leader, sending a full tablet report...
I20260805 22:22:39.239027 30275 ts_tablet_manager.cc:1428] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635: Time spent starting tablet: real 0.027s user 0.013s sys 0.015s
I20260805 22:22:39.239869 30275 tablet_bootstrap.cc:492] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635: Bootstrap starting.
I20260805 22:22:39.248355 30275 tablet_bootstrap.cc:654] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.258515 30275 tablet_bootstrap.cc:492] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635: No bootstrap required, opened a new log
I20260805 22:22:39.259092 30275 ts_tablet_manager.cc:1397] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.013s
I20260805 22:22:39.261479 30277 tablet_bootstrap.cc:492] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c: No bootstrap required, opened a new log
I20260805 22:22:39.262147 30277 ts_tablet_manager.cc:1397] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.019s
I20260805 22:22:39.262240 30275 raft_consensus.cc:348] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.263065 30275 raft_consensus.cc:374] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.263373 30249 heartbeater.cc:503] Master 127.29.21.254:40149 was elected leader, sending a full tablet report...
I20260805 22:22:39.264144 30278 ts_tablet_manager.cc:1428] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27: Time spent starting tablet: real 0.048s user 0.008s sys 0.011s
I20260805 22:22:39.263381 30275 raft_consensus.cc:729] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Initialized, Role: FOLLOWER
I20260805 22:22:39.283562 30278 tablet_bootstrap.cc:492] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27: Bootstrap starting.
I20260805 22:22:39.283344 30275 consensus_queue.cc:260] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.265717 30277 raft_consensus.cc:348] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.284483 30277 raft_consensus.cc:374] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.284794 30277 raft_consensus.cc:729] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607da182484e4982a5d88b345ded418c, State: Initialized, Role: FOLLOWER
I20260805 22:22:39.285578 30277 consensus_queue.cc:260] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.288079 30275 ts_tablet_manager.cc:1428] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635: Time spent starting tablet: real 0.029s user 0.006s sys 0.016s
I20260805 22:22:39.288792 30278 tablet_bootstrap.cc:654] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.289013 30275 tablet_bootstrap.cc:492] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635: Bootstrap starting.
I20260805 22:22:39.295034 30277 ts_tablet_manager.cc:1428] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c: Time spent starting tablet: real 0.032s user 0.012s sys 0.010s
I20260805 22:22:39.295989 30277 tablet_bootstrap.cc:492] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c: Bootstrap starting.
I20260805 22:22:39.302408 30277 tablet_bootstrap.cc:654] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.308104 30275 tablet_bootstrap.cc:654] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.320869 30275 tablet_bootstrap.cc:492] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635: No bootstrap required, opened a new log
I20260805 22:22:39.323052 30275 ts_tablet_manager.cc:1397] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.009s
I20260805 22:22:39.324246 30277 tablet_bootstrap.cc:492] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c: No bootstrap required, opened a new log
I20260805 22:22:39.324724 30277 ts_tablet_manager.cc:1397] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.009s
I20260805 22:22:39.326787 30275 raft_consensus.cc:348] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.328075 30275 raft_consensus.cc:374] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.327600 30277 raft_consensus.cc:348] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.328487 30275 raft_consensus.cc:729] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Initialized, Role: FOLLOWER
I20260805 22:22:39.328642 30277 raft_consensus.cc:374] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.328982 30277 raft_consensus.cc:729] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607da182484e4982a5d88b345ded418c, State: Initialized, Role: FOLLOWER
I20260805 22:22:39.328220 30278 tablet_bootstrap.cc:492] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27: No bootstrap required, opened a new log
I20260805 22:22:39.329807 30278 ts_tablet_manager.cc:1397] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27: Time spent bootstrapping tablet: real 0.046s user 0.018s sys 0.022s
I20260805 22:22:39.329289 30275 consensus_queue.cc:260] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.329789 30277 consensus_queue.cc:260] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.332721 30277 ts_tablet_manager.cc:1428] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c: Time spent starting tablet: real 0.008s user 0.004s sys 0.004s
I20260805 22:22:39.333683 30277 tablet_bootstrap.cc:492] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c: Bootstrap starting.
I20260805 22:22:39.333233 30278 raft_consensus.cc:348] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.334226 30278 raft_consensus.cc:374] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.334610 30278 raft_consensus.cc:729] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Initialized, Role: FOLLOWER
W20260805 22:22:39.334803 30037 tablet.cc:2369] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:22:39.335604 30278 consensus_queue.cc:260] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.338327 30278 ts_tablet_manager.cc:1428] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20260805 22:22:39.338811 30275 ts_tablet_manager.cc:1428] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260805 22:22:39.339370 30278 tablet_bootstrap.cc:492] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27: Bootstrap starting.
I20260805 22:22:39.339759 30275 tablet_bootstrap.cc:492] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635: Bootstrap starting.
I20260805 22:22:39.340036 30277 tablet_bootstrap.cc:654] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.347321 30278 tablet_bootstrap.cc:654] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.348407 30275 tablet_bootstrap.cc:654] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.363415 30278 tablet_bootstrap.cc:492] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27: No bootstrap required, opened a new log
I20260805 22:22:39.363873 30278 ts_tablet_manager.cc:1397] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.004s
I20260805 22:22:39.364284 30277 tablet_bootstrap.cc:492] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c: No bootstrap required, opened a new log
I20260805 22:22:39.364734 30277 ts_tablet_manager.cc:1397] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.007s
I20260805 22:22:39.366729 30278 raft_consensus.cc:348] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.367594 30278 raft_consensus.cc:374] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.367900 30278 raft_consensus.cc:729] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Initialized, Role: FOLLOWER
I20260805 22:22:39.368947 30277 raft_consensus.cc:348] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.368862 30278 consensus_queue.cc:260] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.370007 30277 raft_consensus.cc:374] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.370385 30277 raft_consensus.cc:729] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607da182484e4982a5d88b345ded418c, State: Initialized, Role: FOLLOWER
I20260805 22:22:39.371450 30275 tablet_bootstrap.cc:492] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635: No bootstrap required, opened a new log
I20260805 22:22:39.371201 30277 consensus_queue.cc:260] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.371810 30278 ts_tablet_manager.cc:1428] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260805 22:22:39.371917 30275 ts_tablet_manager.cc:1397] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.005s
I20260805 22:22:39.373766 30277 ts_tablet_manager.cc:1428] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c: Time spent starting tablet: real 0.009s user 0.008s sys 0.002s
I20260805 22:22:39.374840 30277 tablet_bootstrap.cc:492] T a7d64cdb99414419b519f3abb289edb1 P 607da182484e4982a5d88b345ded418c: Bootstrap starting.
I20260805 22:22:39.376446 30275 raft_consensus.cc:348] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.377170 30275 raft_consensus.cc:374] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.377487 30275 raft_consensus.cc:729] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Initialized, Role: FOLLOWER
I20260805 22:22:39.378180 30275 consensus_queue.cc:260] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.380419 30275 ts_tablet_manager.cc:1428] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260805 22:22:39.382156 30278 tablet_bootstrap.cc:492] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27: Bootstrap starting.
I20260805 22:22:39.382270 30277 tablet_bootstrap.cc:654] T a7d64cdb99414419b519f3abb289edb1 P 607da182484e4982a5d88b345ded418c: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.389711 30278 tablet_bootstrap.cc:654] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.392980 30275 tablet_bootstrap.cc:492] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635: Bootstrap starting.
I20260805 22:22:39.402117 30275 tablet_bootstrap.cc:654] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.417157 30277 tablet_bootstrap.cc:492] T a7d64cdb99414419b519f3abb289edb1 P 607da182484e4982a5d88b345ded418c: No bootstrap required, opened a new log
I20260805 22:22:39.417812 30277 ts_tablet_manager.cc:1397] T a7d64cdb99414419b519f3abb289edb1 P 607da182484e4982a5d88b345ded418c: Time spent bootstrapping tablet: real 0.043s user 0.019s sys 0.020s
I20260805 22:22:39.418278 30278 tablet_bootstrap.cc:492] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27: No bootstrap required, opened a new log
I20260805 22:22:39.418738 30278 ts_tablet_manager.cc:1397] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27: Time spent bootstrapping tablet: real 0.037s user 0.015s sys 0.016s
I20260805 22:22:39.420958 30281 raft_consensus.cc:482] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:22:39.421650 30287 raft_consensus.cc:482] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:22:39.421296 30277 raft_consensus.cc:348] T a7d64cdb99414419b519f3abb289edb1 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.421535 30281 raft_consensus.cc:504] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.422181 30287 raft_consensus.cc:504] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:39.425251 30278 raft_consensus.cc:348] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.425944 30281 leader_election.cc:288] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:39.426880 30277 raft_consensus.cc:374] T a7d64cdb99414419b519f3abb289edb1 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.429668 30287 leader_election.cc:288] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069), 607da182484e4982a5d88b345ded418c (127.29.21.193:39793)
I20260805 22:22:39.428964 30277 raft_consensus.cc:729] T a7d64cdb99414419b519f3abb289edb1 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607da182484e4982a5d88b345ded418c, State: Initialized, Role: FOLLOWER
I20260805 22:22:39.432053 30277 consensus_queue.cc:260] T a7d64cdb99414419b519f3abb289edb1 P 607da182484e4982a5d88b345ded418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.435745 30277 ts_tablet_manager.cc:1428] T a7d64cdb99414419b519f3abb289edb1 P 607da182484e4982a5d88b345ded418c: Time spent starting tablet: real 0.018s user 0.005s sys 0.001s
I20260805 22:22:39.436450 30286 raft_consensus.cc:482] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:22:39.436932 30286 raft_consensus.cc:504] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:39.427179 30278 raft_consensus.cc:374] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.439810 30278 raft_consensus.cc:729] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Initialized, Role: FOLLOWER
I20260805 22:22:39.440194 30286 leader_election.cc:288] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287), 607da182484e4982a5d88b345ded418c (127.29.21.193:39793)
I20260805 22:22:39.442056 30277 tablet_bootstrap.cc:492] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c: Bootstrap starting.
I20260805 22:22:39.440702 30286 raft_consensus.cc:482] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:22:39.446516 30286 raft_consensus.cc:504] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.440706 30278 consensus_queue.cc:260] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.517122 30286 leader_election.cc:288] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:39.517158 30278 ts_tablet_manager.cc:1428] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27: Time spent starting tablet: real 0.098s user 0.007s sys 0.001s
I20260805 22:22:39.517750 30275 tablet_bootstrap.cc:492] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635: No bootstrap required, opened a new log
I20260805 22:22:39.518219 30275 ts_tablet_manager.cc:1397] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635: Time spent bootstrapping tablet: real 0.137s user 0.025s sys 0.019s
I20260805 22:22:39.519601 30278 tablet_bootstrap.cc:492] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27: Bootstrap starting.
I20260805 22:22:39.521836 30277 tablet_bootstrap.cc:654] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.539391 30275 raft_consensus.cc:348] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.540953 30275 raft_consensus.cc:374] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.541308 30275 raft_consensus.cc:729] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Initialized, Role: FOLLOWER
I20260805 22:22:39.542127 30275 consensus_queue.cc:260] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.556818 30275 ts_tablet_manager.cc:1428] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635: Time spent starting tablet: real 0.038s user 0.008s sys 0.002s
I20260805 22:22:39.561769 30275 tablet_bootstrap.cc:492] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635: Bootstrap starting.
I20260805 22:22:39.566190 30315 raft_consensus.cc:482] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:22:39.566780 30315 raft_consensus.cc:504] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.568933 30275 tablet_bootstrap.cc:654] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.569096 30315 leader_election.cc:288] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:39.579500 30278 tablet_bootstrap.cc:654] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27: Neither blocks nor log segments found. Creating new log.
W20260805 22:22:39.594976 29918 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260805 22:22:39.602087 30287 raft_consensus.cc:482] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:22:39.602679 30287 raft_consensus.cc:504] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.614991 30287 leader_election.cc:288] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069)
I20260805 22:22:39.632021 30117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777755da0c344319aaa4f32ec55ce02a" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635" is_pre_election: true
I20260805 22:22:39.633615 30117 raft_consensus.cc:2454] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 0.
I20260805 22:22:39.636933 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9ff840de5194b14935947f0ee7ee529" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635" is_pre_election: true
I20260805 22:22:39.639287 30168 leader_election.cc:302] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 1daa858569c149c5a44f92fe8c9bc635; no voters:
W20260805 22:22:39.641784 30168 leader_election.cc:341] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:22:39.642522 30287 raft_consensus.cc:2781] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:22:39.642891 30287 raft_consensus.cc:482] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:22:39.643585 30287 raft_consensus.cc:3037] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:39.644961 30281 raft_consensus.cc:482] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:22:39.646185 30281 raft_consensus.cc:504] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.665951 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6267e585b62945d492c1fbb3c737c38d" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635" is_pre_election: true
I20260805 22:22:39.666728 30116 raft_consensus.cc:2454] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 607da182484e4982a5d88b345ded418c in term 0.
I20260805 22:22:39.668111 30215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6267e585b62945d492c1fbb3c737c38d" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c9abba79e64e92b71def50c1afbe27" is_pre_election: true
I20260805 22:22:39.668468 30277 tablet_bootstrap.cc:492] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c: No bootstrap required, opened a new log
I20260805 22:22:39.668848 30281 leader_election.cc:288] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:39.671339 30281 raft_consensus.cc:482] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:22:39.672206 30281 raft_consensus.cc:504] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.669872 30277 ts_tablet_manager.cc:1397] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c: Time spent bootstrapping tablet: real 0.228s user 0.017s sys 0.015s
I20260805 22:22:39.675704 30281 leader_election.cc:288] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:39.681900 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2230cd3dbabd47788d44be34d3d596c2" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635" is_pre_election: true
I20260805 22:22:39.683228 30116 raft_consensus.cc:2454] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 607da182484e4982a5d88b345ded418c in term 0.
I20260805 22:22:39.683831 30281 raft_consensus.cc:482] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:22:39.694698 30277 raft_consensus.cc:348] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.698848 30277 raft_consensus.cc:374] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.706972 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b676fd9c96e94b01b7e1a9b297d8221f" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635" is_pre_election: true
I20260805 22:22:39.707774 30116 raft_consensus.cc:2454] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 607da182484e4982a5d88b345ded418c in term 0.
I20260805 22:22:39.705081 30215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2230cd3dbabd47788d44be34d3d596c2" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c9abba79e64e92b71def50c1afbe27" is_pre_election: true
I20260805 22:22:39.708519 30278 tablet_bootstrap.cc:492] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27: No bootstrap required, opened a new log
I20260805 22:22:39.704151 30277 raft_consensus.cc:729] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607da182484e4982a5d88b345ded418c, State: Initialized, Role: FOLLOWER
I20260805 22:22:39.698729 30281 raft_consensus.cc:504] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.710105 30278 ts_tablet_manager.cc:1397] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27: Time spent bootstrapping tablet: real 0.191s user 0.000s sys 0.025s
I20260805 22:22:39.710418 30213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b676fd9c96e94b01b7e1a9b297d8221f" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c9abba79e64e92b71def50c1afbe27" is_pre_election: true
I20260805 22:22:39.713510 30277 consensus_queue.cc:260] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.714740 30275 tablet_bootstrap.cc:492] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635: No bootstrap required, opened a new log
I20260805 22:22:39.716132 30275 ts_tablet_manager.cc:1397] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635: Time spent bootstrapping tablet: real 0.155s user 0.022s sys 0.025s
I20260805 22:22:39.717490 30277 ts_tablet_manager.cc:1428] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c: Time spent starting tablet: real 0.038s user 0.005s sys 0.007s
I20260805 22:22:39.718534 30277 tablet_bootstrap.cc:492] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c: Bootstrap starting.
I20260805 22:22:39.718954 30278 raft_consensus.cc:348] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.720243 30278 raft_consensus.cc:374] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.720499 30278 raft_consensus.cc:729] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Initialized, Role: FOLLOWER
I20260805 22:22:39.722321 29944 leader_election.cc:302] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c; no voters:
W20260805 22:22:39.730324 29945 leader_election.cc:341] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:22:39.730784 30281 leader_election.cc:288] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
W20260805 22:22:39.735110 29945 leader_election.cc:341] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:22:39.734911 30278 consensus_queue.cc:260] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
W20260805 22:22:39.735898 29945 leader_election.cc:341] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:22:39.737547 30215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f348c46b854206beab7393137dccb5" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c9abba79e64e92b71def50c1afbe27" is_pre_election: true
I20260805 22:22:39.737186 30275 raft_consensus.cc:348] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.739959 30281 raft_consensus.cc:2781] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:22:39.740134 29997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777755da0c344319aaa4f32ec55ce02a" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607da182484e4982a5d88b345ded418c" is_pre_election: true
I20260805 22:22:39.741008 30281 raft_consensus.cc:482] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:22:39.741362 30281 raft_consensus.cc:3037] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:39.739967 30275 raft_consensus.cc:374] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.741295 29944 leader_election.cc:302] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c; no voters: 09c9abba79e64e92b71def50c1afbe27
I20260805 22:22:39.744560 30275 raft_consensus.cc:729] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Initialized, Role: FOLLOWER
W20260805 22:22:39.745293 29945 leader_election.cc:341] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:22:39.744238 29996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9ff840de5194b14935947f0ee7ee529" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607da182484e4982a5d88b345ded418c" is_pre_election: true
I20260805 22:22:39.749747 29996 raft_consensus.cc:2454] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 0.
W20260805 22:22:39.750099 30169 leader_election.cc:341] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 607da182484e4982a5d88b345ded418c (127.29.21.193:39793): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:22:39.751507 29944 leader_election.cc:302] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c; no voters: 09c9abba79e64e92b71def50c1afbe27
I20260805 22:22:39.756510 30213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b676fd9c96e94b01b7e1a9b297d8221f" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c9abba79e64e92b71def50c1afbe27" is_pre_election: true
I20260805 22:22:39.757423 30278 ts_tablet_manager.cc:1428] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27: Time spent starting tablet: real 0.047s user 0.006s sys 0.012s
I20260805 22:22:39.755021 30329 raft_consensus.cc:2781] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:22:39.751330 30169 leader_election.cc:302] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 607da182484e4982a5d88b345ded418c; no voters: 1daa858569c149c5a44f92fe8c9bc635
W20260805 22:22:39.764125 30064 leader_election.cc:341] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:22:39.761165 30329 raft_consensus.cc:482] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:22:39.754592 30287 raft_consensus.cc:504] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:39.764894 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f348c46b854206beab7393137dccb5" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635" is_pre_election: true
I20260805 22:22:39.759909 30275 consensus_queue.cc:260] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.763882 30278 tablet_bootstrap.cc:492] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27: Bootstrap starting.
I20260805 22:22:39.760775 30212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e86514dd0434e69ae26b33066fc8bc0" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c9abba79e64e92b71def50c1afbe27" is_pre_election: true
I20260805 22:22:39.768329 30329 raft_consensus.cc:3037] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:39.768714 30327 raft_consensus.cc:2781] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:22:39.765168 30215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777755da0c344319aaa4f32ec55ce02a" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c9abba79e64e92b71def50c1afbe27" is_pre_election: true
W20260805 22:22:39.772162 29944 leader_election.cc:341] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:22:39.777788 30064 leader_election.cc:341] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:22:39.777865 29944 leader_election.cc:302] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 607da182484e4982a5d88b345ded418c; no voters: 09c9abba79e64e92b71def50c1afbe27, 1daa858569c149c5a44f92fe8c9bc635
I20260805 22:22:39.777087 30327 raft_consensus.cc:482] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:22:39.778954 30327 raft_consensus.cc:3037] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:39.773957 30330 raft_consensus.cc:2781] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:22:39.780917 30215 raft_consensus.cc:2379] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1daa858569c149c5a44f92fe8c9bc635 in current term 1: Already voted for candidate 09c9abba79e64e92b71def50c1afbe27 in this term.
I20260805 22:22:39.781713 30275 ts_tablet_manager.cc:1428] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635: Time spent starting tablet: real 0.065s user 0.004s sys 0.007s
I20260805 22:22:39.783007 30330 raft_consensus.cc:482] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:22:39.775759 30277 tablet_bootstrap.cc:654] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.784180 30330 raft_consensus.cc:3037] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:39.783878 30275 tablet_bootstrap.cc:492] T a7d64cdb99414419b519f3abb289edb1 P 1daa858569c149c5a44f92fe8c9bc635: Bootstrap starting.
I20260805 22:22:39.789047 30281 raft_consensus.cc:504] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.800606 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777755da0c344319aaa4f32ec55ce02a" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
I20260805 22:22:39.800969 30331 raft_consensus.cc:482] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:22:39.801335 30116 raft_consensus.cc:3037] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:39.806062 29996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777755da0c344319aaa4f32ec55ce02a" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607da182484e4982a5d88b345ded418c"
I20260805 22:22:39.801501 30331 raft_consensus.cc:504] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.804698 30287 leader_election.cc:288] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 election: Requested vote from peers 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069), 607da182484e4982a5d88b345ded418c (127.29.21.193:39793)
W20260805 22:22:39.811240 30169 leader_election.cc:341] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 607da182484e4982a5d88b345ded418c (127.29.21.193:39793): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:22:39.814000 30331 leader_election.cc:288] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069)
I20260805 22:22:39.815090 30117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d7ac208d874bdab1e7ee5d1909814d" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635" is_pre_election: true
I20260805 22:22:39.815336 29996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d7ac208d874bdab1e7ee5d1909814d" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607da182484e4982a5d88b345ded418c" is_pre_election: true
I20260805 22:22:39.816386 29996 raft_consensus.cc:2454] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 0.
I20260805 22:22:39.816501 30281 leader_election.cc:288] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 election: Requested vote from peers 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
W20260805 22:22:39.816990 30168 leader_election.cc:341] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:22:39.818006 30215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6267e585b62945d492c1fbb3c737c38d" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
I20260805 22:22:39.820833 30169 leader_election.cc:302] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 607da182484e4982a5d88b345ded418c; no voters: 1daa858569c149c5a44f92fe8c9bc635
W20260805 22:22:39.821043 29945 leader_election.cc:341] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:22:39.820787 30117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6267e585b62945d492c1fbb3c737c38d" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
I20260805 22:22:39.821791 30117 raft_consensus.cc:3037] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:39.822253 30287 raft_consensus.cc:2781] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:22:39.822675 30287 raft_consensus.cc:482] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:22:39.825733 30287 raft_consensus.cc:3037] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:39.820374 30331 raft_consensus.cc:482] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:22:39.827876 30332 raft_consensus.cc:2726] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:22:39.830416 30116 raft_consensus.cc:2454] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 1.
I20260805 22:22:39.823729 30330 raft_consensus.cc:504] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.829156 30331 raft_consensus.cc:504] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.836287 30168 leader_election.cc:302] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 1daa858569c149c5a44f92fe8c9bc635; no voters: 607da182484e4982a5d88b345ded418c
I20260805 22:22:39.845116 29996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9ff840de5194b14935947f0ee7ee529" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607da182484e4982a5d88b345ded418c"
I20260805 22:22:39.845829 29996 raft_consensus.cc:3037] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:39.846751 29997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d64cdb99414419b519f3abb289edb1" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607da182484e4982a5d88b345ded418c" is_pre_election: true
I20260805 22:22:39.842267 30327 raft_consensus.cc:504] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.843434 30117 raft_consensus.cc:2454] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607da182484e4982a5d88b345ded418c in term 1.
I20260805 22:22:39.844532 30330 leader_election.cc:288] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 election: Requested vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069)
I20260805 22:22:39.848912 29997 raft_consensus.cc:2454] T a7d64cdb99414419b519f3abb289edb1 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 0.
I20260805 22:22:39.852428 30336 raft_consensus.cc:2781] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:22:39.852866 30169 leader_election.cc:302] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 607da182484e4982a5d88b345ded418c; no voters:
I20260805 22:22:39.853539 29944 leader_election.cc:302] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c; no voters: 09c9abba79e64e92b71def50c1afbe27
I20260805 22:22:39.854149 30330 raft_consensus.cc:2781] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:22:39.854770 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d64cdb99414419b519f3abb289edb1" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635" is_pre_election: true
I20260805 22:22:39.854758 30330 raft_consensus.cc:482] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:22:39.854751 30332 raft_consensus.cc:2781] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:22:39.855643 30330 raft_consensus.cc:3037] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:39.855815 30332 raft_consensus.cc:686] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 1 LEADER]: Becoming Leader. State: Replica: 607da182484e4982a5d88b345ded418c, State: Running, Role: LEADER
I20260805 22:22:39.853353 30331 leader_election.cc:288] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069)
I20260805 22:22:39.856043 30117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9ff840de5194b14935947f0ee7ee529" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
I20260805 22:22:39.850744 30329 raft_consensus.cc:504] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.857370 30332 consensus_queue.cc:237] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.853591 30277 tablet_bootstrap.cc:492] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c: No bootstrap required, opened a new log
I20260805 22:22:39.860220 30278 tablet_bootstrap.cc:654] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.860925 30277 ts_tablet_manager.cc:1397] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c: Time spent bootstrapping tablet: real 0.143s user 0.005s sys 0.014s
I20260805 22:22:39.861490 30117 raft_consensus.cc:3037] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:39.868446 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2230cd3dbabd47788d44be34d3d596c2" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
W20260805 22:22:39.862855 30168 leader_election.cc:341] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:22:39.866603 30336 raft_consensus.cc:686] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Becoming Leader. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Running, Role: LEADER
I20260805 22:22:39.870395 30116 raft_consensus.cc:3037] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:39.870641 30277 raft_consensus.cc:348] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:39.884702 30277 raft_consensus.cc:374] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.885654 30277 raft_consensus.cc:729] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607da182484e4982a5d88b345ded418c, State: Initialized, Role: FOLLOWER
I20260805 22:22:39.886124 30327 leader_election.cc:288] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 election: Requested vote from peers 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:39.888391 30287 raft_consensus.cc:504] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.880079 30329 leader_election.cc:288] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 election: Requested vote from peers 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:39.895263 30275 tablet_bootstrap.cc:654] T a7d64cdb99414419b519f3abb289edb1 P 1daa858569c149c5a44f92fe8c9bc635: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.895095 29997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d7ac208d874bdab1e7ee5d1909814d" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607da182484e4982a5d88b345ded418c"
I20260805 22:22:39.896396 30277 consensus_queue.cc:260] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:39.897557 29997 raft_consensus.cc:3037] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:39.895931 30215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2230cd3dbabd47788d44be34d3d596c2" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
I20260805 22:22:39.896730 30287 leader_election.cc:288] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 election: Requested vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069)
I20260805 22:22:39.911230 30277 ts_tablet_manager.cc:1428] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c: Time spent starting tablet: real 0.047s user 0.003s sys 0.004s
I20260805 22:22:39.913246 30277 tablet_bootstrap.cc:492] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c: Bootstrap starting.
I20260805 22:22:39.909072 30215 raft_consensus.cc:3037] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:39.900151 29996 raft_consensus.cc:2454] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 1.
I20260805 22:22:39.909873 30112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d7ac208d874bdab1e7ee5d1909814d" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
I20260805 22:22:39.914654 30212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b676fd9c96e94b01b7e1a9b297d8221f" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
I20260805 22:22:39.918932 29837 catalog_manager.cc:5495] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c reported cstate change: term changed from 0 to 1, leader changed from <none> to 607da182484e4982a5d88b345ded418c (127.29.21.193). New cstate: current_term: 1 leader_uuid: "607da182484e4982a5d88b345ded418c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: UNKNOWN } } }
I20260805 22:22:39.910108 30330 raft_consensus.cc:504] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.913304 29995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777755da0c344319aaa4f32ec55ce02a" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607da182484e4982a5d88b345ded418c" is_pre_election: true
I20260805 22:22:39.914760 30113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b676fd9c96e94b01b7e1a9b297d8221f" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
I20260805 22:22:39.921773 30169 leader_election.cc:302] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 607da182484e4982a5d88b345ded418c; no voters:
I20260805 22:22:39.924451 30113 raft_consensus.cc:3037] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:39.921072 29994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b676fd9c96e94b01b7e1a9b297d8221f" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607da182484e4982a5d88b345ded418c" is_pre_election: true
I20260805 22:22:39.924414 29995 raft_consensus.cc:2454] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 0.
W20260805 22:22:39.923730 29945 leader_election.cc:341] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:22:39.928865 29994 raft_consensus.cc:2379] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1daa858569c149c5a44f92fe8c9bc635 in current term 1: Already voted for candidate 607da182484e4982a5d88b345ded418c in this term.
I20260805 22:22:39.929139 30287 raft_consensus.cc:2781] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:22:39.927392 29996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e86514dd0434e69ae26b33066fc8bc0" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607da182484e4982a5d88b345ded418c" is_pre_election: true
I20260805 22:22:39.929816 30064 leader_election.cc:302] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c; no voters: 09c9abba79e64e92b71def50c1afbe27
I20260805 22:22:39.909252 30336 consensus_queue.cc:237] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:39.934635 30064 leader_election.cc:302] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635; no voters: 09c9abba79e64e92b71def50c1afbe27, 607da182484e4982a5d88b345ded418c
W20260805 22:22:39.926883 30168 leader_election.cc:341] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:22:39.936077 30315 raft_consensus.cc:2741] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260805 22:22:39.936465 30287 raft_consensus.cc:686] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Becoming Leader. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Running, Role: LEADER
I20260805 22:22:39.937500 30287 consensus_queue.cc:237] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
W20260805 22:22:39.943944 30064 leader_election.cc:341] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 607da182484e4982a5d88b345ded418c (127.29.21.193:39793): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:22:39.944435 30064 leader_election.cc:302] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635; no voters: 09c9abba79e64e92b71def50c1afbe27, 607da182484e4982a5d88b345ded418c
I20260805 22:22:39.945371 29996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d64cdb99414419b519f3abb289edb1" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607da182484e4982a5d88b345ded418c"
I20260805 22:22:39.946014 29996 raft_consensus.cc:3037] T a7d64cdb99414419b519f3abb289edb1 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:39.946197 30112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d64cdb99414419b519f3abb289edb1" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
I20260805 22:22:39.945214 30315 raft_consensus.cc:2726] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:22:39.944568 30330 leader_election.cc:288] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 election: Requested vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069)
W20260805 22:22:39.950273 30168 leader_election.cc:341] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:22:39.950292 30113 raft_consensus.cc:2454] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607da182484e4982a5d88b345ded418c in term 1.
I20260805 22:22:39.953476 30286 raft_consensus.cc:2726] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:22:39.955603 29944 leader_election.cc:302] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c; no voters: 09c9abba79e64e92b71def50c1afbe27
I20260805 22:22:39.956566 30329 raft_consensus.cc:2781] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:22:39.949694 30117 raft_consensus.cc:2454] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 1.
I20260805 22:22:39.957050 30329 raft_consensus.cc:686] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 1 LEADER]: Becoming Leader. State: Replica: 607da182484e4982a5d88b345ded418c, State: Running, Role: LEADER
I20260805 22:22:39.958238 30329 consensus_queue.cc:237] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.959081 30116 raft_consensus.cc:2454] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607da182484e4982a5d88b345ded418c in term 1.
I20260805 22:22:39.963245 29833 catalog_manager.cc:5495] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "09c9abba79e64e92b71def50c1afbe27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: UNKNOWN } } }
I20260805 22:22:39.968765 29944 leader_election.cc:302] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c; no voters:
I20260805 22:22:39.970191 30327 raft_consensus.cc:2781] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:22:39.970948 30327 raft_consensus.cc:686] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 1 LEADER]: Becoming Leader. State: Replica: 607da182484e4982a5d88b345ded418c, State: Running, Role: LEADER
I20260805 22:22:39.971787 30277 tablet_bootstrap.cc:654] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:39.972923 30327 consensus_queue.cc:237] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.974885 30278 tablet_bootstrap.cc:492] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27: No bootstrap required, opened a new log
I20260805 22:22:39.975392 30278 ts_tablet_manager.cc:1397] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27: Time spent bootstrapping tablet: real 0.214s user 0.015s sys 0.005s
I20260805 22:22:39.982264 30278 raft_consensus.cc:348] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.983171 30278 raft_consensus.cc:374] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:39.983817 30278 raft_consensus.cc:729] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Initialized, Role: FOLLOWER
I20260805 22:22:39.979362 29997 raft_consensus.cc:2454] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 1.
I20260805 22:22:39.986109 29996 raft_consensus.cc:2454] T a7d64cdb99414419b519f3abb289edb1 P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 1.
I20260805 22:22:39.986799 30169 leader_election.cc:302] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 607da182484e4982a5d88b345ded418c; no voters: 1daa858569c149c5a44f92fe8c9bc635
I20260805 22:22:39.984721 30278 consensus_queue.cc:260] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.988385 30287 raft_consensus.cc:2781] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:22:39.988915 30287 raft_consensus.cc:686] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Becoming Leader. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Running, Role: LEADER
I20260805 22:22:39.988940 30169 leader_election.cc:302] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 607da182484e4982a5d88b345ded418c; no voters: 1daa858569c149c5a44f92fe8c9bc635
I20260805 22:22:39.989785 30336 raft_consensus.cc:2781] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:22:39.989950 30287 consensus_queue.cc:237] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.990424 30336 raft_consensus.cc:686] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Becoming Leader. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Running, Role: LEADER
I20260805 22:22:39.991425 30336 consensus_queue.cc:237] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:39.991926 30278 ts_tablet_manager.cc:1428] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27: Time spent starting tablet: real 0.016s user 0.005s sys 0.003s
I20260805 22:22:39.996099 30278 tablet_bootstrap.cc:492] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27: Bootstrap starting.
I20260805 22:22:40.001250 30275 tablet_bootstrap.cc:492] T a7d64cdb99414419b519f3abb289edb1 P 1daa858569c149c5a44f92fe8c9bc635: No bootstrap required, opened a new log
I20260805 22:22:40.004796 30275 ts_tablet_manager.cc:1397] T a7d64cdb99414419b519f3abb289edb1 P 1daa858569c149c5a44f92fe8c9bc635: Time spent bootstrapping tablet: real 0.221s user 0.016s sys 0.005s
I20260805 22:22:40.010594 30215 raft_consensus.cc:2454] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607da182484e4982a5d88b345ded418c in term 1.
I20260805 22:22:40.009728 30275 raft_consensus.cc:348] T a7d64cdb99414419b519f3abb289edb1 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.011487 30275 raft_consensus.cc:374] T a7d64cdb99414419b519f3abb289edb1 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:40.013597 30275 raft_consensus.cc:729] T a7d64cdb99414419b519f3abb289edb1 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Initialized, Role: FOLLOWER
I20260805 22:22:40.015179 30275 consensus_queue.cc:260] T a7d64cdb99414419b519f3abb289edb1 P 1daa858569c149c5a44f92fe8c9bc635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.021200 30278 tablet_bootstrap.cc:654] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:40.021871 30275 ts_tablet_manager.cc:1428] T a7d64cdb99414419b519f3abb289edb1 P 1daa858569c149c5a44f92fe8c9bc635: Time spent starting tablet: real 0.017s user 0.006s sys 0.002s
I20260805 22:22:40.023681 30275 tablet_bootstrap.cc:492] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635: Bootstrap starting.
I20260805 22:22:40.030892 30277 tablet_bootstrap.cc:492] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c: No bootstrap required, opened a new log
I20260805 22:22:40.031935 30277 ts_tablet_manager.cc:1397] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c: Time spent bootstrapping tablet: real 0.119s user 0.012s sys 0.006s
I20260805 22:22:40.039742 30277 raft_consensus.cc:348] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.040530 30277 raft_consensus.cc:374] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:40.040819 30277 raft_consensus.cc:729] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 607da182484e4982a5d88b345ded418c, State: Initialized, Role: FOLLOWER
I20260805 22:22:40.042966 30277 consensus_queue.cc:260] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.044973 30277 ts_tablet_manager.cc:1428] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260805 22:22:40.047683 30275 tablet_bootstrap.cc:654] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:40.049536 30278 tablet_bootstrap.cc:492] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27: No bootstrap required, opened a new log
I20260805 22:22:40.050094 30278 ts_tablet_manager.cc:1397] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27: Time spent bootstrapping tablet: real 0.054s user 0.021s sys 0.013s
I20260805 22:22:40.053268 30278 raft_consensus.cc:348] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.053939 30278 raft_consensus.cc:374] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:40.054239 30278 raft_consensus.cc:729] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Initialized, Role: FOLLOWER
I20260805 22:22:40.055127 30278 consensus_queue.cc:260] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.058080 30278 ts_tablet_manager.cc:1428] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27: Time spent starting tablet: real 0.008s user 0.005s sys 0.003s
I20260805 22:22:40.059022 30278 tablet_bootstrap.cc:492] T a8f348c46b854206beab7393137dccb5 P 09c9abba79e64e92b71def50c1afbe27: Bootstrap starting.
I20260805 22:22:40.066601 30275 tablet_bootstrap.cc:492] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635: No bootstrap required, opened a new log
I20260805 22:22:40.067113 30275 ts_tablet_manager.cc:1397] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.004s
I20260805 22:22:40.069362 30278 tablet_bootstrap.cc:654] T a8f348c46b854206beab7393137dccb5 P 09c9abba79e64e92b71def50c1afbe27: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:40.070338 30275 raft_consensus.cc:348] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.071132 30275 raft_consensus.cc:374] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:40.071401 30275 raft_consensus.cc:729] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Initialized, Role: FOLLOWER
I20260805 22:22:40.072098 30275 consensus_queue.cc:260] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.075557 30275 ts_tablet_manager.cc:1428] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635: Time spent starting tablet: real 0.008s user 0.004s sys 0.003s
I20260805 22:22:40.076428 30275 tablet_bootstrap.cc:492] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635: Bootstrap starting.
I20260805 22:22:40.082686 30275 tablet_bootstrap.cc:654] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:40.088016 29833 catalog_manager.cc:5495] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c reported cstate change: term changed from 0 to 1, leader changed from <none> to 607da182484e4982a5d88b345ded418c (127.29.21.193). New cstate: current_term: 1 leader_uuid: "607da182484e4982a5d88b345ded418c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: UNKNOWN } } }
I20260805 22:22:40.089896 29833 catalog_manager.cc:5495] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c reported cstate change: term changed from 0 to 1, leader changed from <none> to 607da182484e4982a5d88b345ded418c (127.29.21.193). New cstate: current_term: 1 leader_uuid: "607da182484e4982a5d88b345ded418c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: UNKNOWN } } }
I20260805 22:22:40.095547 30275 tablet_bootstrap.cc:492] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635: No bootstrap required, opened a new log
I20260805 22:22:40.095770 30278 tablet_bootstrap.cc:492] T a8f348c46b854206beab7393137dccb5 P 09c9abba79e64e92b71def50c1afbe27: No bootstrap required, opened a new log
I20260805 22:22:40.096159 30275 ts_tablet_manager.cc:1397] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.004s
I20260805 22:22:40.096266 30278 ts_tablet_manager.cc:1397] T a8f348c46b854206beab7393137dccb5 P 09c9abba79e64e92b71def50c1afbe27: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.012s
I20260805 22:22:40.098884 30275 raft_consensus.cc:348] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.099658 30275 raft_consensus.cc:374] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:40.099975 30275 raft_consensus.cc:729] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Initialized, Role: FOLLOWER
I20260805 22:22:40.100818 30275 consensus_queue.cc:260] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.099201 30278 raft_consensus.cc:348] T a8f348c46b854206beab7393137dccb5 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.103020 30275 ts_tablet_manager.cc:1428] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260805 22:22:40.102986 30278 raft_consensus.cc:374] T a8f348c46b854206beab7393137dccb5 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:40.103482 30278 raft_consensus.cc:729] T a8f348c46b854206beab7393137dccb5 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Initialized, Role: FOLLOWER
I20260805 22:22:40.102838 29837 catalog_manager.cc:5495] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "09c9abba79e64e92b71def50c1afbe27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: HEALTHY } } }
I20260805 22:22:40.104111 30278 consensus_queue.cc:260] T a8f348c46b854206beab7393137dccb5 P 09c9abba79e64e92b71def50c1afbe27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.104334 29837 catalog_manager.cc:5495] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "09c9abba79e64e92b71def50c1afbe27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: HEALTHY } } }
I20260805 22:22:40.105636 29837 catalog_manager.cc:5495] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "09c9abba79e64e92b71def50c1afbe27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: HEALTHY } } }
I20260805 22:22:40.106055 30278 ts_tablet_manager.cc:1428] T a8f348c46b854206beab7393137dccb5 P 09c9abba79e64e92b71def50c1afbe27: Time spent starting tablet: real 0.009s user 0.007s sys 0.001s
I20260805 22:22:40.106962 30278 tablet_bootstrap.cc:492] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27: Bootstrap starting.
I20260805 22:22:40.113517 30278 tablet_bootstrap.cc:654] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27: Neither blocks nor log segments found. Creating new log.
I20260805 22:22:40.115067 30286 raft_consensus.cc:482] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:22:40.115557 30286 raft_consensus.cc:504] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.117493 30286 leader_election.cc:288] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:40.118314 29996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f348c46b854206beab7393137dccb5" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607da182484e4982a5d88b345ded418c" is_pre_election: true
I20260805 22:22:40.118670 30215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f348c46b854206beab7393137dccb5" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c9abba79e64e92b71def50c1afbe27" is_pre_election: true
I20260805 22:22:40.119029 29996 raft_consensus.cc:2454] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 0.
I20260805 22:22:40.119266 30215 raft_consensus.cc:2454] T a8f348c46b854206beab7393137dccb5 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 0.
I20260805 22:22:40.120421 30064 leader_election.cc:302] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 1daa858569c149c5a44f92fe8c9bc635; no voters:
I20260805 22:22:40.121349 30278 tablet_bootstrap.cc:492] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27: No bootstrap required, opened a new log
I20260805 22:22:40.121836 30278 ts_tablet_manager.cc:1397] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20260805 22:22:40.123703 30286 raft_consensus.cc:2781] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:22:40.124114 30286 raft_consensus.cc:482] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:22:40.124516 30286 raft_consensus.cc:3037] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:40.124418 30278 raft_consensus.cc:348] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.125142 30278 raft_consensus.cc:374] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:22:40.125433 30278 raft_consensus.cc:729] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Initialized, Role: FOLLOWER
I20260805 22:22:40.126551 30278 consensus_queue.cc:260] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.128834 30278 ts_tablet_manager.cc:1428] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260805 22:22:40.133507 30286 raft_consensus.cc:504] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.135593 30286 leader_election.cc:288] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 1 election: Requested vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:40.136693 29996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f348c46b854206beab7393137dccb5" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607da182484e4982a5d88b345ded418c"
I20260805 22:22:40.136837 30215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a8f348c46b854206beab7393137dccb5" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
I20260805 22:22:40.137184 29996 raft_consensus.cc:3037] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:40.137467 30215 raft_consensus.cc:3037] T a8f348c46b854206beab7393137dccb5 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:40.142462 30215 raft_consensus.cc:2454] T a8f348c46b854206beab7393137dccb5 P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 1.
I20260805 22:22:40.143570 30064 leader_election.cc:302] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 1daa858569c149c5a44f92fe8c9bc635; no voters:
I20260805 22:22:40.143998 29996 raft_consensus.cc:2454] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 1.
I20260805 22:22:40.144331 30286 raft_consensus.cc:2781] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:22:40.144932 30286 raft_consensus.cc:686] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [term 1 LEADER]: Becoming Leader. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Running, Role: LEADER
I20260805 22:22:40.145728 30286 consensus_queue.cc:237] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.153980 29833 catalog_manager.cc:5495] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "1daa858569c149c5a44f92fe8c9bc635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: UNKNOWN } } }
I20260805 22:22:40.175201 30336 raft_consensus.cc:482] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:22:40.175781 30336 raft_consensus.cc:504] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.177994 30336 leader_election.cc:288] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069)
I20260805 22:22:40.178879 29996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e86514dd0434e69ae26b33066fc8bc0" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607da182484e4982a5d88b345ded418c" is_pre_election: true
I20260805 22:22:40.179080 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e86514dd0434e69ae26b33066fc8bc0" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635" is_pre_election: true
I20260805 22:22:40.179438 29996 raft_consensus.cc:2454] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 0.
I20260805 22:22:40.179759 30116 raft_consensus.cc:2454] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 0.
I20260805 22:22:40.180668 30169 leader_election.cc:302] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 607da182484e4982a5d88b345ded418c; no voters:
I20260805 22:22:40.181568 30336 raft_consensus.cc:2781] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:22:40.181959 30336 raft_consensus.cc:482] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:22:40.182219 30336 raft_consensus.cc:3037] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:40.187825 30336 raft_consensus.cc:504] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.189607 30336 leader_election.cc:288] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 election: Requested vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069)
I20260805 22:22:40.190433 29996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e86514dd0434e69ae26b33066fc8bc0" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607da182484e4982a5d88b345ded418c"
I20260805 22:22:40.190824 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e86514dd0434e69ae26b33066fc8bc0" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
I20260805 22:22:40.191057 29996 raft_consensus.cc:3037] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:40.191390 30116 raft_consensus.cc:3037] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:40.197750 29996 raft_consensus.cc:2454] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 1.
I20260805 22:22:40.199048 30169 leader_election.cc:302] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 607da182484e4982a5d88b345ded418c; no voters:
I20260805 22:22:40.200006 30336 raft_consensus.cc:2781] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:22:40.197748 30116 raft_consensus.cc:2454] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 1.
I20260805 22:22:40.201259 30336 raft_consensus.cc:686] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Becoming Leader. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Running, Role: LEADER
I20260805 22:22:40.202761 30336 consensus_queue.cc:237] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.210906 29833 catalog_manager.cc:5495] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "09c9abba79e64e92b71def50c1afbe27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: HEALTHY } } }
I20260805 22:22:40.290378 30336 raft_consensus.cc:482] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:22:40.290987 30336 raft_consensus.cc:504] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.293427 30336 leader_election.cc:288] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069)
I20260805 22:22:40.294574 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6267e585b62945d492c1fbb3c737c38d" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635" is_pre_election: true
I20260805 22:22:40.295457 30116 raft_consensus.cc:2379] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09c9abba79e64e92b71def50c1afbe27 in current term 1: Already voted for candidate 607da182484e4982a5d88b345ded418c in this term.
I20260805 22:22:40.295521 29996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6267e585b62945d492c1fbb3c737c38d" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607da182484e4982a5d88b345ded418c" is_pre_election: true
I20260805 22:22:40.297731 30169 leader_election.cc:302] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27; no voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c
I20260805 22:22:40.298434 30336 raft_consensus.cc:3037] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:40.305405 30336 raft_consensus.cc:2726] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:22:40.311302 30327 consensus_queue.cc:1035] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:22:40.315392 30327 consensus_queue.cc:1035] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:22:40.337360 30287 consensus_queue.cc:1035] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:22:40.366173 30327 consensus_queue.cc:1035] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:22:40.374604 30117 raft_consensus.cc:3037] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:40.378290 30336 consensus_queue.cc:1035] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:22:40.384562 30336 raft_consensus.cc:482] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:22:40.385208 30336 raft_consensus.cc:504] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.389164 29996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b676fd9c96e94b01b7e1a9b297d8221f" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607da182484e4982a5d88b345ded418c" is_pre_election: true
I20260805 22:22:40.399406 30336 leader_election.cc:288] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069)
I20260805 22:22:40.399821 30113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b676fd9c96e94b01b7e1a9b297d8221f" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1daa858569c149c5a44f92fe8c9bc635" is_pre_election: true
I20260805 22:22:40.401172 30113 raft_consensus.cc:2379] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09c9abba79e64e92b71def50c1afbe27 in current term 1: Already voted for candidate 607da182484e4982a5d88b345ded418c in this term.
I20260805 22:22:40.403074 30168 leader_election.cc:302] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27; no voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c
I20260805 22:22:40.405616 30341 consensus_queue.cc:1035] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:22:40.406255 30336 raft_consensus.cc:3037] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:40.419452 30327 consensus_queue.cc:1035] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:22:40.421360 30336 raft_consensus.cc:2726] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:22:40.439756 30327 consensus_queue.cc:1035] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:22:40.451985 30336 consensus_queue.cc:1035] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:22:40.479904 30286 raft_consensus.cc:482] T a7d64cdb99414419b519f3abb289edb1 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:22:40.480827 30286 raft_consensus.cc:504] T a7d64cdb99414419b519f3abb289edb1 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.487680 29996 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d64cdb99414419b519f3abb289edb1" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "607da182484e4982a5d88b345ded418c" is_pre_election: true
I20260805 22:22:40.489120 29996 raft_consensus.cc:2379] T a7d64cdb99414419b519f3abb289edb1 P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1daa858569c149c5a44f92fe8c9bc635 in current term 1: Already voted for candidate 09c9abba79e64e92b71def50c1afbe27 in this term.
I20260805 22:22:40.494325 30286 leader_election.cc:288] T a7d64cdb99414419b519f3abb289edb1 P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:40.500849 30215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7d64cdb99414419b519f3abb289edb1" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09c9abba79e64e92b71def50c1afbe27" is_pre_election: true
I20260805 22:22:40.508719 30116 raft_consensus.cc:3037] T a7d64cdb99414419b519f3abb289edb1 P 1daa858569c149c5a44f92fe8c9bc635 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:40.508766 30064 leader_election.cc:302] T a7d64cdb99414419b519f3abb289edb1 P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635; no voters: 09c9abba79e64e92b71def50c1afbe27, 607da182484e4982a5d88b345ded418c
I20260805 22:22:40.509563 30329 consensus_queue.cc:1035] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:22:40.522248 29996 raft_consensus.cc:3037] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:22:40.528585 30286 raft_consensus.cc:2726] T a7d64cdb99414419b519f3abb289edb1 P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:22:40.539871 30341 consensus_queue.cc:1035] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:22:40.549850 30341 consensus_queue.cc:1035] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:22:40.556632 30287 consensus_queue.cc:1035] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:22:40.587904 30330 consensus_queue.cc:1035] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:22:40.610317 30341 consensus_queue.cc:1035] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:22:40.631568 30215 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "777755da0c344319aaa4f32ec55ce02a"
dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
mode: GRACEFUL
new_leader_uuid: "607da182484e4982a5d88b345ded418c"
from {username='slave'} at 127.0.0.1:50234
I20260805 22:22:40.632455 30215 raft_consensus.cc:595] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Received request to transfer leadership to TS 607da182484e4982a5d88b345ded418c
I20260805 22:22:40.649712 30215 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9ff840de5194b14935947f0ee7ee529"
dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
mode: GRACEFUL
new_leader_uuid: "607da182484e4982a5d88b345ded418c"
from {username='slave'} at 127.0.0.1:50234
I20260805 22:22:40.650370 30215 raft_consensus.cc:595] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Received request to transfer leadership to TS 607da182484e4982a5d88b345ded418c
I20260805 22:22:40.656862 30341 consensus_queue.cc:1035] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:22:40.679224 29994 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6267e585b62945d492c1fbb3c737c38d"
dest_uuid: "607da182484e4982a5d88b345ded418c"
mode: GRACEFUL
new_leader_uuid: "1daa858569c149c5a44f92fe8c9bc635"
from {username='slave'} at 127.0.0.1:45138
I20260805 22:22:40.679762 29994 raft_consensus.cc:595] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 1 LEADER]: Received request to transfer leadership to TS 1daa858569c149c5a44f92fe8c9bc635
I20260805 22:22:40.684971 30215 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "05d7ac208d874bdab1e7ee5d1909814d"
dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
mode: GRACEFUL
new_leader_uuid: "1daa858569c149c5a44f92fe8c9bc635"
from {username='slave'} at 127.0.0.1:50234
I20260805 22:22:40.685571 30215 raft_consensus.cc:595] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Received request to transfer leadership to TS 1daa858569c149c5a44f92fe8c9bc635
I20260805 22:22:40.695457 30215 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e86514dd0434e69ae26b33066fc8bc0"
dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
mode: GRACEFUL
new_leader_uuid: "1daa858569c149c5a44f92fe8c9bc635"
from {username='slave'} at 127.0.0.1:50234
I20260805 22:22:40.696094 30215 raft_consensus.cc:595] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Received request to transfer leadership to TS 1daa858569c149c5a44f92fe8c9bc635
I20260805 22:22:40.703791 30286 consensus_queue.cc:1035] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:22:40.711422 30286 consensus_queue.cc:1035] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:22:40.770056 30330 raft_consensus.cc:979] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27: : Instructing follower 607da182484e4982a5d88b345ded418c to start an election
I20260805 22:22:40.770604 30341 raft_consensus.cc:1067] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Signalling peer 607da182484e4982a5d88b345ded418c to start an election
I20260805 22:22:40.772341 29997 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9ff840de5194b14935947f0ee7ee529"
dest_uuid: "607da182484e4982a5d88b345ded418c"
from {username='slave'} at 127.0.0.1:45112
I20260805 22:22:40.772905 29997 raft_consensus.cc:482] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:22:40.773216 29997 raft_consensus.cc:3037] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:40.778302 29997 raft_consensus.cc:504] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.780153 29997 leader_election.cc:288] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 2 election: Requested vote from peers 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:40.781210 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9ff840de5194b14935947f0ee7ee529" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
I20260805 22:22:40.781643 30215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9ff840de5194b14935947f0ee7ee529" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
I20260805 22:22:40.782032 30116 raft_consensus.cc:3037] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:40.782377 30215 raft_consensus.cc:3032] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:22:40.782774 30215 raft_consensus.cc:729] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Running, Role: LEADER
I20260805 22:22:40.783636 30215 consensus_queue.cc:260] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.784795 30215 raft_consensus.cc:3037] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:40.788899 30116 raft_consensus.cc:2454] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607da182484e4982a5d88b345ded418c in term 2.
I20260805 22:22:40.789265 30215 raft_consensus.cc:2454] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607da182484e4982a5d88b345ded418c in term 2.
I20260805 22:22:40.790189 29944 leader_election.cc:302] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c; no voters:
I20260805 22:22:40.791041 30327 raft_consensus.cc:2781] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:22:40.791426 30327 raft_consensus.cc:686] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 2 LEADER]: Becoming Leader. State: Replica: 607da182484e4982a5d88b345ded418c, State: Running, Role: LEADER
I20260805 22:22:40.792166 30327 consensus_queue.cc:237] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.798523 29833 catalog_manager.cc:5495] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c reported cstate change: term changed from 1 to 2, leader changed from 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195) to 607da182484e4982a5d88b345ded418c (127.29.21.193). New cstate: current_term: 2 leader_uuid: "607da182484e4982a5d88b345ded418c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: UNKNOWN } } }
I20260805 22:22:40.827160 30389 raft_consensus.cc:979] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c: : Instructing follower 1daa858569c149c5a44f92fe8c9bc635 to start an election
I20260805 22:22:40.827680 30327 raft_consensus.cc:1067] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 1 LEADER]: Signalling peer 1daa858569c149c5a44f92fe8c9bc635 to start an election
I20260805 22:22:40.829589 30116 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6267e585b62945d492c1fbb3c737c38d"
dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
from {username='slave'} at 127.0.0.1:53034
I20260805 22:22:40.830231 30116 raft_consensus.cc:482] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:22:40.830649 30116 raft_consensus.cc:3037] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:40.838406 30116 raft_consensus.cc:504] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.840607 30116 leader_election.cc:288] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 2 election: Requested vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:40.841538 29997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6267e585b62945d492c1fbb3c737c38d" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "607da182484e4982a5d88b345ded418c"
I20260805 22:22:40.842028 30215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6267e585b62945d492c1fbb3c737c38d" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
I20260805 22:22:40.842280 29997 raft_consensus.cc:3032] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:22:40.842625 29997 raft_consensus.cc:729] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 607da182484e4982a5d88b345ded418c, State: Running, Role: LEADER
I20260805 22:22:40.842639 30215 raft_consensus.cc:3037] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:40.843518 29997 consensus_queue.cc:260] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.844771 29997 raft_consensus.cc:3037] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:40.849720 29997 raft_consensus.cc:2454] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 2.
I20260805 22:22:40.849723 30215 raft_consensus.cc:2454] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 2.
I20260805 22:22:40.850602 30212 raft_consensus.cc:1226] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Rejecting Update request from peer 607da182484e4982a5d88b345ded418c for earlier term 1. Current term is 2. Ops: []
I20260805 22:22:40.850992 30064 leader_election.cc:302] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c; no voters:
I20260805 22:22:40.851850 30404 raft_consensus.cc:2781] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:22:40.852296 30404 raft_consensus.cc:686] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [term 2 LEADER]: Becoming Leader. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Running, Role: LEADER
I20260805 22:22:40.853305 30404 consensus_queue.cc:237] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:40.862516 29833 catalog_manager.cc:5495] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 reported cstate change: term changed from 1 to 2, leader changed from 607da182484e4982a5d88b345ded418c (127.29.21.193) to 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "1daa858569c149c5a44f92fe8c9bc635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: UNKNOWN } } }
I20260805 22:22:40.956526 30330 raft_consensus.cc:979] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27: : Instructing follower 607da182484e4982a5d88b345ded418c to start an election
I20260805 22:22:40.957026 30341 raft_consensus.cc:1067] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Signalling peer 607da182484e4982a5d88b345ded418c to start an election
I20260805 22:22:40.958798 29997 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "777755da0c344319aaa4f32ec55ce02a"
dest_uuid: "607da182484e4982a5d88b345ded418c"
from {username='slave'} at 127.0.0.1:45112
I20260805 22:22:40.959311 29997 raft_consensus.cc:482] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:22:40.959616 29997 raft_consensus.cc:3037] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:40.964032 29997 raft_consensus.cc:504] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:40.965677 29997 leader_election.cc:288] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 2 election: Requested vote from peers 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287), 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069)
I20260805 22:22:40.966823 30212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777755da0c344319aaa4f32ec55ce02a" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
I20260805 22:22:40.966938 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777755da0c344319aaa4f32ec55ce02a" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
I20260805 22:22:40.967496 30212 raft_consensus.cc:3032] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:22:40.967617 30116 raft_consensus.cc:3037] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:40.967864 30212 raft_consensus.cc:729] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Running, Role: LEADER
I20260805 22:22:40.968861 30212 consensus_queue.cc:260] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:40.970192 30212 raft_consensus.cc:3037] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:40.972923 30116 raft_consensus.cc:2454] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607da182484e4982a5d88b345ded418c in term 2.
I20260805 22:22:40.973943 29944 leader_election.cc:302] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c; no voters:
I20260805 22:22:40.974635 30327 raft_consensus.cc:2781] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:22:40.975021 30327 raft_consensus.cc:686] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 2 LEADER]: Becoming Leader. State: Replica: 607da182484e4982a5d88b345ded418c, State: Running, Role: LEADER
I20260805 22:22:40.975835 30327 consensus_queue.cc:237] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:40.976766 30212 raft_consensus.cc:2454] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607da182484e4982a5d88b345ded418c in term 2.
I20260805 22:22:40.982784 29833 catalog_manager.cc:5495] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c reported cstate change: term changed from 1 to 2, leader changed from 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195) to 607da182484e4982a5d88b345ded418c (127.29.21.193). New cstate: current_term: 2 leader_uuid: "607da182484e4982a5d88b345ded418c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: HEALTHY } } }
I20260805 22:22:41.180358 30330 raft_consensus.cc:979] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27: : Instructing follower 1daa858569c149c5a44f92fe8c9bc635 to start an election
I20260805 22:22:41.180817 30330 raft_consensus.cc:1067] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Signalling peer 1daa858569c149c5a44f92fe8c9bc635 to start an election
I20260805 22:22:41.182157 30116 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "05d7ac208d874bdab1e7ee5d1909814d"
dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
from {username='slave'} at 127.0.0.1:53044
I20260805 22:22:41.182624 30116 raft_consensus.cc:482] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:22:41.182910 30116 raft_consensus.cc:3037] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:41.187932 30116 raft_consensus.cc:504] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:41.189677 30116 leader_election.cc:288] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 2 election: Requested vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:41.190393 29997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d7ac208d874bdab1e7ee5d1909814d" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "607da182484e4982a5d88b345ded418c"
I20260805 22:22:41.190654 30212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d7ac208d874bdab1e7ee5d1909814d" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
I20260805 22:22:41.191040 29997 raft_consensus.cc:3037] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:41.191213 30212 raft_consensus.cc:3032] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:22:41.191553 30212 raft_consensus.cc:729] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Running, Role: LEADER
I20260805 22:22:41.192276 30212 consensus_queue.cc:260] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:41.193519 30212 raft_consensus.cc:3037] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:41.197654 29997 raft_consensus.cc:2454] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 2.
I20260805 22:22:41.198781 30064 leader_election.cc:302] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c; no voters:
I20260805 22:22:41.199482 30404 raft_consensus.cc:2781] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:22:41.199559 30212 raft_consensus.cc:2454] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 2.
I20260805 22:22:41.200007 30404 raft_consensus.cc:686] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [term 2 LEADER]: Becoming Leader. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Running, Role: LEADER
I20260805 22:22:41.200817 30404 consensus_queue.cc:237] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:41.206821 29833 catalog_manager.cc:5495] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 reported cstate change: term changed from 1 to 2, leader changed from 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195) to 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "1daa858569c149c5a44f92fe8c9bc635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: UNKNOWN } } }
I20260805 22:22:41.217835 30341 raft_consensus.cc:979] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27: : Instructing follower 1daa858569c149c5a44f92fe8c9bc635 to start an election
I20260805 22:22:41.218305 30330 raft_consensus.cc:1067] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Signalling peer 1daa858569c149c5a44f92fe8c9bc635 to start an election
I20260805 22:22:41.219687 30116 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e86514dd0434e69ae26b33066fc8bc0"
dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
from {username='slave'} at 127.0.0.1:53044
I20260805 22:22:41.220204 30116 raft_consensus.cc:482] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:22:41.220513 30116 raft_consensus.cc:3037] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:41.224977 30116 raft_consensus.cc:504] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:41.226516 30116 leader_election.cc:288] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 2 election: Requested vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:41.227500 29997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e86514dd0434e69ae26b33066fc8bc0" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "607da182484e4982a5d88b345ded418c"
I20260805 22:22:41.227759 30212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e86514dd0434e69ae26b33066fc8bc0" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
I20260805 22:22:41.228127 29997 raft_consensus.cc:3037] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:41.228425 30212 raft_consensus.cc:3032] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:22:41.228754 30212 raft_consensus.cc:729] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Running, Role: LEADER
I20260805 22:22:41.229465 30212 consensus_queue.cc:260] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:41.230384 30212 raft_consensus.cc:3037] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:41.233050 29997 raft_consensus.cc:2454] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 2.
I20260805 22:22:41.234076 30064 leader_election.cc:302] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c; no voters:
I20260805 22:22:41.234884 30404 raft_consensus.cc:2781] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:22:41.235180 30212 raft_consensus.cc:2454] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 2.
I20260805 22:22:41.235304 30404 raft_consensus.cc:686] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 2 LEADER]: Becoming Leader. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Running, Role: LEADER
I20260805 22:22:41.236589 30404 consensus_queue.cc:237] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:41.242614 29833 catalog_manager.cc:5495] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 reported cstate change: term changed from 1 to 2, leader changed from 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195) to 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "1daa858569c149c5a44f92fe8c9bc635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: UNKNOWN } } }
I20260805 22:22:41.317636 30116 raft_consensus.cc:1261] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Refusing update from remote peer 607da182484e4982a5d88b345ded418c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:22:41.319123 30327 consensus_queue.cc:1035] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:22:41.328620 30212 raft_consensus.cc:1261] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Refusing update from remote peer 607da182484e4982a5d88b345ded418c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:22:41.330313 30327 consensus_queue.cc:1035] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:22:41.346057 29997 raft_consensus.cc:1261] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Refusing update from remote peer 1daa858569c149c5a44f92fe8c9bc635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:22:41.347282 30404 consensus_queue.cc:1035] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:22:41.355103 30212 raft_consensus.cc:1261] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Refusing update from remote peer 1daa858569c149c5a44f92fe8c9bc635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:22:41.356731 30418 consensus_queue.cc:1035] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:22:41.472061 30116 raft_consensus.cc:1261] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Refusing update from remote peer 607da182484e4982a5d88b345ded418c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:22:41.473335 30327 consensus_queue.cc:1035] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:22:41.480487 30212 raft_consensus.cc:1261] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Refusing update from remote peer 607da182484e4982a5d88b345ded418c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:22:41.482450 30329 consensus_queue.cc:1035] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:22:41.727306 29997 raft_consensus.cc:1261] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Refusing update from remote peer 1daa858569c149c5a44f92fe8c9bc635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:22:41.728468 30418 consensus_queue.cc:1035] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:22:41.737399 30212 raft_consensus.cc:1261] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Refusing update from remote peer 1daa858569c149c5a44f92fe8c9bc635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:22:41.738658 30418 consensus_queue.cc:1035] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:22:41.739030 29994 raft_consensus.cc:1261] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Refusing update from remote peer 1daa858569c149c5a44f92fe8c9bc635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:22:41.740669 30418 consensus_queue.cc:1035] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:22:41.757869 30212 raft_consensus.cc:1261] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Refusing update from remote peer 1daa858569c149c5a44f92fe8c9bc635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:22:41.761433 30425 consensus_queue.cc:1035] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:22:43.732542 29783 auto_leader_rebalancer-test.cc:269] The leader distribution is
I20260805 22:22:43.733498 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:22:43.751361 29994 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2230cd3dbabd47788d44be34d3d596c2"
dest_uuid: "607da182484e4982a5d88b345ded418c"
mode: GRACEFUL
new_leader_uuid: "09c9abba79e64e92b71def50c1afbe27"
from {username='slave'} at 127.0.0.1:45154
I20260805 22:22:43.751901 29994 raft_consensus.cc:595] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 1 LEADER]: Received request to transfer leadership to TS 09c9abba79e64e92b71def50c1afbe27
I20260805 22:22:43.763867 30116 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e86514dd0434e69ae26b33066fc8bc0"
dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
mode: GRACEFUL
new_leader_uuid: "09c9abba79e64e92b71def50c1afbe27"
from {username='slave'} at 127.0.0.1:53058
I20260805 22:22:43.764509 30116 raft_consensus.cc:595] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 2 LEADER]: Received request to transfer leadership to TS 09c9abba79e64e92b71def50c1afbe27
I20260805 22:22:43.765661 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260805 22:22:43.766127 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:22:44.217363 30389 raft_consensus.cc:979] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c: : Instructing follower 09c9abba79e64e92b71def50c1afbe27 to start an election
I20260805 22:22:44.217792 30430 raft_consensus.cc:1067] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 1 LEADER]: Signalling peer 09c9abba79e64e92b71def50c1afbe27 to start an election
I20260805 22:22:44.219367 30215 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2230cd3dbabd47788d44be34d3d596c2"
dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
from {username='slave'} at 127.0.0.1:50222
I20260805 22:22:44.219837 30215 raft_consensus.cc:482] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:22:44.220114 30215 raft_consensus.cc:3037] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:44.224723 30215 raft_consensus.cc:504] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:44.226402 30215 leader_election.cc:288] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 2 election: Requested vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069)
I20260805 22:22:44.227366 29994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2230cd3dbabd47788d44be34d3d596c2" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "607da182484e4982a5d88b345ded418c"
I20260805 22:22:44.227468 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2230cd3dbabd47788d44be34d3d596c2" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
I20260805 22:22:44.227926 29994 raft_consensus.cc:3032] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:22:44.228019 30116 raft_consensus.cc:3037] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:44.228219 29994 raft_consensus.cc:729] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 607da182484e4982a5d88b345ded418c, State: Running, Role: LEADER
I20260805 22:22:44.228931 29994 consensus_queue.cc:260] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:44.229982 29994 raft_consensus.cc:3037] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:44.233049 30116 raft_consensus.cc:2454] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 2.
I20260805 22:22:44.234226 30168 leader_election.cc:302] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 1daa858569c149c5a44f92fe8c9bc635; no voters:
I20260805 22:22:44.234612 29994 raft_consensus.cc:2454] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 2.
I20260805 22:22:44.234992 30330 raft_consensus.cc:2781] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:22:44.236210 30330 raft_consensus.cc:686] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 2 LEADER]: Becoming Leader. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Running, Role: LEADER
I20260805 22:22:44.236922 30330 consensus_queue.cc:237] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:44.243634 30433 raft_consensus.cc:979] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635: : Instructing follower 09c9abba79e64e92b71def50c1afbe27 to start an election
I20260805 22:22:44.244163 30425 raft_consensus.cc:1067] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 2 LEADER]: Signalling peer 09c9abba79e64e92b71def50c1afbe27 to start an election
I20260805 22:22:44.245286 29837 catalog_manager.cc:5495] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 reported cstate change: term changed from 1 to 2, leader changed from 607da182484e4982a5d88b345ded418c (127.29.21.193) to 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "09c9abba79e64e92b71def50c1afbe27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: HEALTHY } } }
I20260805 22:22:44.245888 30215 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e86514dd0434e69ae26b33066fc8bc0"
dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
from {username='slave'} at 127.0.0.1:50230
I20260805 22:22:44.246392 30215 raft_consensus.cc:482] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:22:44.246692 30215 raft_consensus.cc:3037] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:22:44.251713 30215 raft_consensus.cc:504] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:44.253741 30215 leader_election.cc:288] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 3 election: Requested vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069)
I20260805 22:22:44.254622 29997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e86514dd0434e69ae26b33066fc8bc0" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "607da182484e4982a5d88b345ded418c"
I20260805 22:22:44.254765 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e86514dd0434e69ae26b33066fc8bc0" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
I20260805 22:22:44.255254 29997 raft_consensus.cc:3037] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:22:44.255322 30116 raft_consensus.cc:3032] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:22:44.255867 30116 raft_consensus.cc:729] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Running, Role: LEADER
I20260805 22:22:44.256482 30116 consensus_queue.cc:260] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:44.257221 30116 raft_consensus.cc:3037] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:22:44.260950 30116 raft_consensus.cc:2454] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 3.
I20260805 22:22:44.261940 30168 leader_election.cc:302] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 1daa858569c149c5a44f92fe8c9bc635; no voters:
I20260805 22:22:44.262778 30330 raft_consensus.cc:2781] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:22:44.262874 29997 raft_consensus.cc:2454] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 3.
I20260805 22:22:44.263371 30330 raft_consensus.cc:686] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 3 LEADER]: Becoming Leader. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Running, Role: LEADER
I20260805 22:22:44.264258 30330 consensus_queue.cc:237] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:44.271466 29837 catalog_manager.cc:5495] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 reported cstate change: term changed from 2 to 3, leader changed from 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194) to 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195). New cstate: current_term: 3 leader_uuid: "09c9abba79e64e92b71def50c1afbe27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: HEALTHY } } }
I20260805 22:22:44.752430 30116 raft_consensus.cc:1261] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Refusing update from remote peer 09c9abba79e64e92b71def50c1afbe27: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:22:44.753652 30330 consensus_queue.cc:1035] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:22:44.762605 29997 raft_consensus.cc:1261] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Refusing update from remote peer 09c9abba79e64e92b71def50c1afbe27: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:22:44.764108 30330 consensus_queue.cc:1035] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:22:44.767024 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:22:44.777616 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:22:44.778146 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:22:44.796692 30116 raft_consensus.cc:1261] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Refusing update from remote peer 09c9abba79e64e92b71def50c1afbe27: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:22:44.797904 30330 consensus_queue.cc:1035] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:22:44.807101 29997 raft_consensus.cc:1261] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Refusing update from remote peer 09c9abba79e64e92b71def50c1afbe27: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:22:44.808905 30330 consensus_queue.cc:1035] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:22:45.779515 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:22:45.785815 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:22:45.786233 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:22:46.787102 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:22:46.792160 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:22:46.792588 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:22:47.793438 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:22:47.798295 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:22:47.798710 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:22:48.799487 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:22:48.804171 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:22:48.804579 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:22:49.805366 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:22:49.810748 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:22:49.811239 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:22:50.812012 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:22:50.816934 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:22:50.817358 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:22:51.818233 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:22:51.823179 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:22:51.823601 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:22:52.824369 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:22:52.829066 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:22:52.829511 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:22:53.855504 29783 auto_leader_rebalancer-test.cc:288] The leader distribution is
I20260805 22:22:53.859705 29994 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "777755da0c344319aaa4f32ec55ce02a"
dest_uuid: "607da182484e4982a5d88b345ded418c"
mode: GRACEFUL
new_leader_uuid: "1daa858569c149c5a44f92fe8c9bc635"
from {username='slave'} at 127.0.0.1:45138
I20260805 22:22:53.860631 29994 raft_consensus.cc:595] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 2 LEADER]: Received request to transfer leadership to TS 1daa858569c149c5a44f92fe8c9bc635
I20260805 22:22:53.865211 29994 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9ff840de5194b14935947f0ee7ee529"
dest_uuid: "607da182484e4982a5d88b345ded418c"
mode: GRACEFUL
new_leader_uuid: "1daa858569c149c5a44f92fe8c9bc635"
from {username='slave'} at 127.0.0.1:45138
I20260805 22:22:53.865772 29994 raft_consensus.cc:595] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 2 LEADER]: Received request to transfer leadership to TS 1daa858569c149c5a44f92fe8c9bc635
I20260805 22:22:53.869863 29994 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b676fd9c96e94b01b7e1a9b297d8221f"
dest_uuid: "607da182484e4982a5d88b345ded418c"
mode: GRACEFUL
new_leader_uuid: "1daa858569c149c5a44f92fe8c9bc635"
from {username='slave'} at 127.0.0.1:45138
I20260805 22:22:53.870332 29994 raft_consensus.cc:595] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 1 LEADER]: Received request to transfer leadership to TS 1daa858569c149c5a44f92fe8c9bc635
I20260805 22:22:53.874739 30212 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2230cd3dbabd47788d44be34d3d596c2"
dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
mode: GRACEFUL
new_leader_uuid: "1daa858569c149c5a44f92fe8c9bc635"
from {username='slave'} at 127.0.0.1:50234
I20260805 22:22:53.875280 30212 raft_consensus.cc:595] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 2 LEADER]: Received request to transfer leadership to TS 1daa858569c149c5a44f92fe8c9bc635
I20260805 22:22:53.884723 30212 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3e86514dd0434e69ae26b33066fc8bc0"
dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
mode: GRACEFUL
new_leader_uuid: "1daa858569c149c5a44f92fe8c9bc635"
from {username='slave'} at 127.0.0.1:50234
I20260805 22:22:53.885241 30212 raft_consensus.cc:595] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 3 LEADER]: Received request to transfer leadership to TS 1daa858569c149c5a44f92fe8c9bc635
I20260805 22:22:54.071489 30470 raft_consensus.cc:979] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c: : Instructing follower 1daa858569c149c5a44f92fe8c9bc635 to start an election
I20260805 22:22:54.071959 30470 raft_consensus.cc:1067] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 2 LEADER]: Signalling peer 1daa858569c149c5a44f92fe8c9bc635 to start an election
I20260805 22:22:54.073649 30112 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9ff840de5194b14935947f0ee7ee529"
dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
from {username='slave'} at 127.0.0.1:53034
I20260805 22:22:54.074208 30112 raft_consensus.cc:482] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:22:54.074638 30112 raft_consensus.cc:3037] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:22:54.079720 30112 raft_consensus.cc:504] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:54.081821 30112 leader_election.cc:288] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 3 election: Requested vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:54.083189 30212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9ff840de5194b14935947f0ee7ee529" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
I20260805 22:22:54.083817 30212 raft_consensus.cc:3037] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:22:54.083969 29994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9ff840de5194b14935947f0ee7ee529" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "607da182484e4982a5d88b345ded418c"
I20260805 22:22:54.084481 29994 raft_consensus.cc:3032] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:22:54.084710 29994 raft_consensus.cc:729] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 607da182484e4982a5d88b345ded418c, State: Running, Role: LEADER
I20260805 22:22:54.085359 29994 consensus_queue.cc:260] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:54.086405 29994 raft_consensus.cc:3037] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:22:54.088681 30212 raft_consensus.cc:2454] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 3.
I20260805 22:22:54.089831 30064 leader_election.cc:302] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 1daa858569c149c5a44f92fe8c9bc635; no voters:
I20260805 22:22:54.090631 30463 raft_consensus.cc:2781] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:22:54.091106 29994 raft_consensus.cc:2454] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 3.
I20260805 22:22:54.091621 30463 raft_consensus.cc:686] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 LEADER]: Becoming Leader. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Running, Role: LEADER
I20260805 22:22:54.092517 30463 consensus_queue.cc:237] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:54.099974 29835 catalog_manager.cc:5495] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 reported cstate change: term changed from 2 to 3, leader changed from 607da182484e4982a5d88b345ded418c (127.29.21.193) to 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194). New cstate: current_term: 3 leader_uuid: "1daa858569c149c5a44f92fe8c9bc635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: UNKNOWN } } }
I20260805 22:22:54.124251 30466 raft_consensus.cc:979] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c: : Instructing follower 1daa858569c149c5a44f92fe8c9bc635 to start an election
I20260805 22:22:54.124686 30470 raft_consensus.cc:1067] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 1 LEADER]: Signalling peer 1daa858569c149c5a44f92fe8c9bc635 to start an election
I20260805 22:22:54.126121 30112 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b676fd9c96e94b01b7e1a9b297d8221f"
dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
from {username='slave'} at 127.0.0.1:53034
I20260805 22:22:54.126573 30112 raft_consensus.cc:482] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:22:54.126844 30112 raft_consensus.cc:3037] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:54.131582 30112 raft_consensus.cc:504] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:54.133219 30112 leader_election.cc:288] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 2 election: Requested vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:54.134080 29994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b676fd9c96e94b01b7e1a9b297d8221f" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "607da182484e4982a5d88b345ded418c"
I20260805 22:22:54.134333 30212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b676fd9c96e94b01b7e1a9b297d8221f" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
I20260805 22:22:54.134701 29994 raft_consensus.cc:3032] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:22:54.134856 30212 raft_consensus.cc:3037] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:54.135053 29994 raft_consensus.cc:729] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 607da182484e4982a5d88b345ded418c, State: Running, Role: LEADER
I20260805 22:22:54.136224 30330 raft_consensus.cc:979] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27: : Instructing follower 1daa858569c149c5a44f92fe8c9bc635 to start an election
I20260805 22:22:54.135851 29994 consensus_queue.cc:260] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:54.136722 30480 raft_consensus.cc:1067] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 2 LEADER]: Signalling peer 1daa858569c149c5a44f92fe8c9bc635 to start an election
I20260805 22:22:54.137233 29994 raft_consensus.cc:3037] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:22:54.138182 30112 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2230cd3dbabd47788d44be34d3d596c2"
dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
from {username='slave'} at 127.0.0.1:53044
I20260805 22:22:54.138621 30112 raft_consensus.cc:482] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:22:54.138912 30112 raft_consensus.cc:3037] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:22:54.141665 30212 raft_consensus.cc:2454] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 2.
I20260805 22:22:54.142973 30064 leader_election.cc:302] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 1daa858569c149c5a44f92fe8c9bc635; no voters:
I20260805 22:22:54.143822 30463 raft_consensus.cc:2781] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:22:54.143886 29994 raft_consensus.cc:2454] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 2.
I20260805 22:22:54.143647 30112 raft_consensus.cc:504] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:54.144464 30463 raft_consensus.cc:686] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 2 LEADER]: Becoming Leader. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Running, Role: LEADER
I20260805 22:22:54.145877 30112 leader_election.cc:288] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 3 election: Requested vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:54.145485 30463 consensus_queue.cc:237] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:54.146875 29994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2230cd3dbabd47788d44be34d3d596c2" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "607da182484e4982a5d88b345ded418c"
I20260805 22:22:54.147390 29994 raft_consensus.cc:3037] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:22:54.149874 30212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2230cd3dbabd47788d44be34d3d596c2" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
I20260805 22:22:54.150570 30212 raft_consensus.cc:3032] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:22:54.150890 30212 raft_consensus.cc:729] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Running, Role: LEADER
I20260805 22:22:54.151696 30212 consensus_queue.cc:260] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:54.152686 30212 raft_consensus.cc:3037] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:22:54.153316 29994 raft_consensus.cc:2454] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 3.
I20260805 22:22:54.153496 29835 catalog_manager.cc:5495] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 reported cstate change: term changed from 1 to 2, leader changed from 607da182484e4982a5d88b345ded418c (127.29.21.193) to 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "1daa858569c149c5a44f92fe8c9bc635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: UNKNOWN } } }
I20260805 22:22:54.154381 30064 leader_election.cc:302] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c; no voters:
I20260805 22:22:54.155197 30463 raft_consensus.cc:2781] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:22:54.155643 30463 raft_consensus.cc:686] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 LEADER]: Becoming Leader. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Running, Role: LEADER
I20260805 22:22:54.156389 30463 consensus_queue.cc:237] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:54.158314 30212 raft_consensus.cc:2454] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 3.
I20260805 22:22:54.165686 29835 catalog_manager.cc:5495] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 reported cstate change: term changed from 2 to 3, leader changed from 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195) to 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194). New cstate: current_term: 3 leader_uuid: "1daa858569c149c5a44f92fe8c9bc635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: UNKNOWN } } }
I20260805 22:22:54.356736 30330 raft_consensus.cc:979] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27: : Instructing follower 1daa858569c149c5a44f92fe8c9bc635 to start an election
I20260805 22:22:54.357187 30480 raft_consensus.cc:1067] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 3 LEADER]: Signalling peer 1daa858569c149c5a44f92fe8c9bc635 to start an election
I20260805 22:22:54.358744 30112 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3e86514dd0434e69ae26b33066fc8bc0"
dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
from {username='slave'} at 127.0.0.1:53044
I20260805 22:22:54.359248 30112 raft_consensus.cc:482] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:22:54.359565 30112 raft_consensus.cc:3037] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:22:54.383072 30112 raft_consensus.cc:504] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:54.384781 30112 leader_election.cc:288] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 4 election: Requested vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:54.385741 29994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e86514dd0434e69ae26b33066fc8bc0" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "607da182484e4982a5d88b345ded418c"
I20260805 22:22:54.385962 30212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e86514dd0434e69ae26b33066fc8bc0" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
I20260805 22:22:54.386284 29994 raft_consensus.cc:3037] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:22:54.386430 30212 raft_consensus.cc:3032] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:22:54.386785 30212 raft_consensus.cc:729] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Running, Role: LEADER
I20260805 22:22:54.387645 30212 consensus_queue.cc:260] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:54.388478 30212 raft_consensus.cc:3037] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:22:54.391505 29994 raft_consensus.cc:2454] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 4.
I20260805 22:22:54.392763 30064 leader_election.cc:302] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c; no voters:
I20260805 22:22:54.392812 30212 raft_consensus.cc:2454] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 4.
I20260805 22:22:54.393689 30463 raft_consensus.cc:2781] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:22:54.394316 30466 raft_consensus.cc:979] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c: : Instructing follower 1daa858569c149c5a44f92fe8c9bc635 to start an election
I20260805 22:22:54.394315 30463 raft_consensus.cc:686] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 4 LEADER]: Becoming Leader. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Running, Role: LEADER
I20260805 22:22:54.394886 30470 raft_consensus.cc:1067] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 2 LEADER]: Signalling peer 1daa858569c149c5a44f92fe8c9bc635 to start an election
I20260805 22:22:54.395262 30463 consensus_queue.cc:237] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:54.396337 30112 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "777755da0c344319aaa4f32ec55ce02a"
dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
from {username='slave'} at 127.0.0.1:53034
I20260805 22:22:54.396895 30112 raft_consensus.cc:482] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:22:54.397265 30112 raft_consensus.cc:3037] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:22:54.402928 29835 catalog_manager.cc:5495] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 reported cstate change: term changed from 3 to 4, leader changed from 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195) to 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194). New cstate: current_term: 4 leader_uuid: "1daa858569c149c5a44f92fe8c9bc635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: UNKNOWN } } }
I20260805 22:22:54.457487 30112 raft_consensus.cc:504] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:54.459801 30112 leader_election.cc:288] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 3 election: Requested vote from peers 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287), 607da182484e4982a5d88b345ded418c (127.29.21.193:39793)
I20260805 22:22:54.461043 30212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777755da0c344319aaa4f32ec55ce02a" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
I20260805 22:22:54.461081 29994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777755da0c344319aaa4f32ec55ce02a" candidate_uuid: "1daa858569c149c5a44f92fe8c9bc635" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "607da182484e4982a5d88b345ded418c"
I20260805 22:22:54.461748 30212 raft_consensus.cc:3037] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:22:54.461803 29994 raft_consensus.cc:3032] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:22:54.462152 29994 raft_consensus.cc:729] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 607da182484e4982a5d88b345ded418c, State: Running, Role: LEADER
I20260805 22:22:54.462913 29994 consensus_queue.cc:260] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:54.464133 29994 raft_consensus.cc:3037] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:22:54.493849 30212 raft_consensus.cc:2454] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 3.
I20260805 22:22:54.493849 29994 raft_consensus.cc:2454] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1daa858569c149c5a44f92fe8c9bc635 in term 3.
I20260805 22:22:54.495319 30064 leader_election.cc:302] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c; no voters:
I20260805 22:22:54.496075 30463 raft_consensus.cc:2781] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:22:54.496573 30463 raft_consensus.cc:686] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 3 LEADER]: Becoming Leader. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Running, Role: LEADER
I20260805 22:22:54.497298 30463 consensus_queue.cc:237] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:54.504473 29835 catalog_manager.cc:5495] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 reported cstate change: term changed from 2 to 3, leader changed from 607da182484e4982a5d88b345ded418c (127.29.21.193) to 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194). New cstate: current_term: 3 leader_uuid: "1daa858569c149c5a44f92fe8c9bc635" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: UNKNOWN } } }
I20260805 22:22:54.519546 30212 raft_consensus.cc:1261] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Refusing update from remote peer 1daa858569c149c5a44f92fe8c9bc635: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:22:54.520637 30463 consensus_queue.cc:1035] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:22:54.530553 29994 raft_consensus.cc:1261] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Refusing update from remote peer 1daa858569c149c5a44f92fe8c9bc635: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:22:54.532102 30463 consensus_queue.cc:1035] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:22:54.575711 29994 raft_consensus.cc:1261] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Refusing update from remote peer 1daa858569c149c5a44f92fe8c9bc635: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:22:54.577121 30471 consensus_queue.cc:1035] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:22:54.588471 30212 raft_consensus.cc:1261] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Refusing update from remote peer 1daa858569c149c5a44f92fe8c9bc635: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:22:54.589790 30471 consensus_queue.cc:1035] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:22:54.642107 29994 raft_consensus.cc:1261] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Refusing update from remote peer 1daa858569c149c5a44f92fe8c9bc635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:22:54.643514 30471 consensus_queue.cc:1035] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:22:54.653112 30212 raft_consensus.cc:1261] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Refusing update from remote peer 1daa858569c149c5a44f92fe8c9bc635: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:22:54.654820 30499 consensus_queue.cc:1035] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:22:54.862008 30212 raft_consensus.cc:1261] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 4 FOLLOWER]: Refusing update from remote peer 1daa858569c149c5a44f92fe8c9bc635: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:22:54.863140 30499 consensus_queue.cc:1035] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:22:54.872390 29994 raft_consensus.cc:1261] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c [term 4 FOLLOWER]: Refusing update from remote peer 1daa858569c149c5a44f92fe8c9bc635: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:22:54.873965 30471 consensus_queue.cc:1035] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:22:55.030696 29994 raft_consensus.cc:1261] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Refusing update from remote peer 1daa858569c149c5a44f92fe8c9bc635: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:22:55.032646 30471 consensus_queue.cc:1035] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:22:55.043574 30215 raft_consensus.cc:1261] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Refusing update from remote peer 1daa858569c149c5a44f92fe8c9bc635: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:22:55.044987 30463 consensus_queue.cc:1035] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:22:56.918861 29783 auto_leader_rebalancer-test.cc:303] The leader distribution is
I20260805 22:22:56.919643 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:22:56.926079 30112 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "05d7ac208d874bdab1e7ee5d1909814d"
dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
mode: GRACEFUL
new_leader_uuid: "607da182484e4982a5d88b345ded418c"
from {username='slave'} at 127.0.0.1:53058
I20260805 22:22:56.926779 30112 raft_consensus.cc:595] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [term 2 LEADER]: Received request to transfer leadership to TS 607da182484e4982a5d88b345ded418c
I20260805 22:22:56.929340 30112 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2230cd3dbabd47788d44be34d3d596c2"
dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
mode: GRACEFUL
new_leader_uuid: "607da182484e4982a5d88b345ded418c"
from {username='slave'} at 127.0.0.1:53058
I20260805 22:22:56.929847 30112 raft_consensus.cc:595] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 LEADER]: Received request to transfer leadership to TS 607da182484e4982a5d88b345ded418c
I20260805 22:22:56.931944 30112 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "777755da0c344319aaa4f32ec55ce02a"
dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
mode: GRACEFUL
new_leader_uuid: "09c9abba79e64e92b71def50c1afbe27"
from {username='slave'} at 127.0.0.1:53058
I20260805 22:22:56.932387 30112 raft_consensus.cc:595] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 3 LEADER]: Received request to transfer leadership to TS 09c9abba79e64e92b71def50c1afbe27
I20260805 22:22:56.934405 30112 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b676fd9c96e94b01b7e1a9b297d8221f"
dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
mode: GRACEFUL
new_leader_uuid: "607da182484e4982a5d88b345ded418c"
from {username='slave'} at 127.0.0.1:53058
I20260805 22:22:56.934846 30112 raft_consensus.cc:595] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 2 LEADER]: Received request to transfer leadership to TS 607da182484e4982a5d88b345ded418c
I20260805 22:22:56.936865 30112 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c9ff840de5194b14935947f0ee7ee529"
dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
mode: GRACEFUL
new_leader_uuid: "09c9abba79e64e92b71def50c1afbe27"
from {username='slave'} at 127.0.0.1:53058
I20260805 22:22:56.937310 30112 raft_consensus.cc:595] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 LEADER]: Received request to transfer leadership to TS 09c9abba79e64e92b71def50c1afbe27
I20260805 22:22:56.938256 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260805 22:22:56.938711 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:22:56.967796 30463 raft_consensus.cc:979] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635: : Instructing follower 607da182484e4982a5d88b345ded418c to start an election
I20260805 22:22:56.968374 30499 raft_consensus.cc:1067] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [term 2 LEADER]: Signalling peer 607da182484e4982a5d88b345ded418c to start an election
I20260805 22:22:56.969969 29997 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "05d7ac208d874bdab1e7ee5d1909814d"
dest_uuid: "607da182484e4982a5d88b345ded418c"
from {username='slave'} at 127.0.0.1:45124
I20260805 22:22:56.970552 29997 raft_consensus.cc:482] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:22:56.970953 29997 raft_consensus.cc:3037] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:22:56.976490 29997 raft_consensus.cc:504] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:56.978438 29997 leader_election.cc:288] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 3 election: Requested vote from peers 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:56.979660 30212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d7ac208d874bdab1e7ee5d1909814d" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
I20260805 22:22:56.979609 30112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05d7ac208d874bdab1e7ee5d1909814d" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
I20260805 22:22:56.980284 30212 raft_consensus.cc:3037] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:22:56.980370 30112 raft_consensus.cc:3032] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:22:56.980696 30112 raft_consensus.cc:729] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Running, Role: LEADER
I20260805 22:22:56.981432 30112 consensus_queue.cc:260] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:56.982625 30112 raft_consensus.cc:3037] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:22:56.985134 30212 raft_consensus.cc:2454] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607da182484e4982a5d88b345ded418c in term 3.
I20260805 22:22:56.986389 29945 leader_election.cc:302] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 607da182484e4982a5d88b345ded418c; no voters:
I20260805 22:22:56.988083 30509 raft_consensus.cc:2781] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:22:56.988106 30112 raft_consensus.cc:2454] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607da182484e4982a5d88b345ded418c in term 3.
I20260805 22:22:56.989158 30509 raft_consensus.cc:686] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [term 3 LEADER]: Becoming Leader. State: Replica: 607da182484e4982a5d88b345ded418c, State: Running, Role: LEADER
I20260805 22:22:56.990056 30509 consensus_queue.cc:237] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:56.996474 29837 catalog_manager.cc:5495] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c reported cstate change: term changed from 2 to 3, leader changed from 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194) to 607da182484e4982a5d88b345ded418c (127.29.21.193). New cstate: current_term: 3 leader_uuid: "607da182484e4982a5d88b345ded418c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: UNKNOWN } } }
I20260805 22:22:57.002456 30463 raft_consensus.cc:979] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635: : Instructing follower 09c9abba79e64e92b71def50c1afbe27 to start an election
I20260805 22:22:57.002923 30499 raft_consensus.cc:1067] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 LEADER]: Signalling peer 09c9abba79e64e92b71def50c1afbe27 to start an election
I20260805 22:22:57.004540 30212 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c9ff840de5194b14935947f0ee7ee529"
dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
from {username='slave'} at 127.0.0.1:50230
I20260805 22:22:57.004971 30212 raft_consensus.cc:482] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:22:57.005218 30212 raft_consensus.cc:3037] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:22:57.011117 30212 raft_consensus.cc:504] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:57.013118 30212 leader_election.cc:288] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 4 election: Requested vote from peers 607da182484e4982a5d88b345ded418c (127.29.21.193:39793), 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069)
I20260805 22:22:57.013969 29997 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9ff840de5194b14935947f0ee7ee529" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "607da182484e4982a5d88b345ded418c"
I20260805 22:22:57.014175 30112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9ff840de5194b14935947f0ee7ee529" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
I20260805 22:22:57.014447 29997 raft_consensus.cc:3037] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:22:57.014667 30112 raft_consensus.cc:3032] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:22:57.014946 30112 raft_consensus.cc:729] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Running, Role: LEADER
I20260805 22:22:57.015522 30112 consensus_queue.cc:260] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:57.016443 30112 raft_consensus.cc:3037] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:22:57.018625 29997 raft_consensus.cc:2454] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 4.
I20260805 22:22:57.019722 30169 leader_election.cc:302] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 607da182484e4982a5d88b345ded418c; no voters:
I20260805 22:22:57.020515 30480 raft_consensus.cc:2781] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:22:57.021191 30112 raft_consensus.cc:2454] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 4.
I20260805 22:22:57.021788 30480 raft_consensus.cc:686] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 4 LEADER]: Becoming Leader. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Running, Role: LEADER
I20260805 22:22:57.022707 30480 consensus_queue.cc:237] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:57.029816 29837 catalog_manager.cc:5495] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 reported cstate change: term changed from 3 to 4, leader changed from 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194) to 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195). New cstate: current_term: 4 leader_uuid: "09c9abba79e64e92b71def50c1afbe27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: HEALTHY } } }
I20260805 22:22:57.122231 30463 raft_consensus.cc:979] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635: : Instructing follower 607da182484e4982a5d88b345ded418c to start an election
I20260805 22:22:57.122709 30499 raft_consensus.cc:1067] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 LEADER]: Signalling peer 607da182484e4982a5d88b345ded418c to start an election
I20260805 22:22:57.124243 29997 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2230cd3dbabd47788d44be34d3d596c2"
dest_uuid: "607da182484e4982a5d88b345ded418c"
from {username='slave'} at 127.0.0.1:45124
I20260805 22:22:57.124703 29997 raft_consensus.cc:482] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:22:57.124990 29997 raft_consensus.cc:3037] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:22:57.129863 30463 raft_consensus.cc:979] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635: : Instructing follower 607da182484e4982a5d88b345ded418c to start an election
I20260805 22:22:57.130326 30499 raft_consensus.cc:1067] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 2 LEADER]: Signalling peer 607da182484e4982a5d88b345ded418c to start an election
I20260805 22:22:57.130474 29997 raft_consensus.cc:504] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:57.132007 29994 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b676fd9c96e94b01b7e1a9b297d8221f"
dest_uuid: "607da182484e4982a5d88b345ded418c"
from {username='slave'} at 127.0.0.1:45124
I20260805 22:22:57.132159 29997 leader_election.cc:288] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 4 election: Requested vote from peers 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:57.132571 29994 raft_consensus.cc:482] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:22:57.132915 29994 raft_consensus.cc:3037] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:22:57.133033 30212 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2230cd3dbabd47788d44be34d3d596c2" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
I20260805 22:22:57.133759 30212 raft_consensus.cc:3037] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:22:57.133193 30112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2230cd3dbabd47788d44be34d3d596c2" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
I20260805 22:22:57.135907 30112 raft_consensus.cc:3032] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:22:57.136279 30112 raft_consensus.cc:729] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Running, Role: LEADER
I20260805 22:22:57.137506 30112 consensus_queue.cc:260] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:57.138829 30463 raft_consensus.cc:979] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635: : Instructing follower 09c9abba79e64e92b71def50c1afbe27 to start an election
I20260805 22:22:57.139335 30463 raft_consensus.cc:1067] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 3 LEADER]: Signalling peer 09c9abba79e64e92b71def50c1afbe27 to start an election
I20260805 22:22:57.139297 30112 raft_consensus.cc:3037] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:22:57.139678 29994 raft_consensus.cc:504] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:57.141207 30213 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "777755da0c344319aaa4f32ec55ce02a"
dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
from {username='slave'} at 127.0.0.1:50230
I20260805 22:22:57.141674 30212 raft_consensus.cc:2454] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607da182484e4982a5d88b345ded418c in term 4.
I20260805 22:22:57.141834 30213 raft_consensus.cc:482] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:22:57.142215 30213 raft_consensus.cc:3037] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:22:57.142720 30116 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b676fd9c96e94b01b7e1a9b297d8221f" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
I20260805 22:22:57.143257 30116 raft_consensus.cc:3032] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:22:57.143535 30116 raft_consensus.cc:729] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Running, Role: LEADER
I20260805 22:22:57.143921 30210 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b676fd9c96e94b01b7e1a9b297d8221f" candidate_uuid: "607da182484e4982a5d88b345ded418c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "09c9abba79e64e92b71def50c1afbe27"
I20260805 22:22:57.144562 30210 raft_consensus.cc:3037] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:22:57.144255 30116 consensus_queue.cc:260] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:57.145438 30116 raft_consensus.cc:3037] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:22:57.146229 30112 raft_consensus.cc:2454] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607da182484e4982a5d88b345ded418c in term 4.
I20260805 22:22:57.147523 29994 leader_election.cc:288] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 3 election: Requested vote from peers 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069), 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195:39287)
I20260805 22:22:57.147553 29944 leader_election.cc:302] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c; no voters:
I20260805 22:22:57.149061 30509 raft_consensus.cc:2781] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:22:57.149587 30509 raft_consensus.cc:686] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 4 LEADER]: Becoming Leader. State: Replica: 607da182484e4982a5d88b345ded418c, State: Running, Role: LEADER
I20260805 22:22:57.149438 30213 raft_consensus.cc:504] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:57.150390 30509 consensus_queue.cc:237] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:57.152173 30112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777755da0c344319aaa4f32ec55ce02a" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "1daa858569c149c5a44f92fe8c9bc635"
I20260805 22:22:57.151904 30116 raft_consensus.cc:2454] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607da182484e4982a5d88b345ded418c in term 3.
I20260805 22:22:57.152961 30215 raft_consensus.cc:1226] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 4 FOLLOWER]: Rejecting Update request from peer 1daa858569c149c5a44f92fe8c9bc635 for earlier term 3. Current term is 4. Ops: []
I20260805 22:22:57.153306 30112 raft_consensus.cc:3032] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:22:57.155272 30210 raft_consensus.cc:2454] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 607da182484e4982a5d88b345ded418c in term 3.
I20260805 22:22:57.155507 29944 leader_election.cc:302] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1daa858569c149c5a44f92fe8c9bc635, 607da182484e4982a5d88b345ded418c; no voters:
I20260805 22:22:57.155761 30213 leader_election.cc:288] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 4 election: Requested vote from peers 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194:39069), 607da182484e4982a5d88b345ded418c (127.29.21.193:39793)
I20260805 22:22:57.154311 30112 raft_consensus.cc:729] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 1daa858569c149c5a44f92fe8c9bc635, State: Running, Role: LEADER
I20260805 22:22:57.156976 30511 raft_consensus.cc:2781] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:22:57.157403 29994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "777755da0c344319aaa4f32ec55ce02a" candidate_uuid: "09c9abba79e64e92b71def50c1afbe27" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "607da182484e4982a5d88b345ded418c"
I20260805 22:22:57.158007 29994 raft_consensus.cc:3037] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:22:57.157588 30112 consensus_queue.cc:260] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:57.159013 30112 raft_consensus.cc:3037] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:22:57.159761 30511 raft_consensus.cc:686] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 3 LEADER]: Becoming Leader. State: Replica: 607da182484e4982a5d88b345ded418c, State: Running, Role: LEADER
I20260805 22:22:57.160647 30511 consensus_queue.cc:237] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } }
I20260805 22:22:57.163374 29837 catalog_manager.cc:5495] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c reported cstate change: term changed from 3 to 4, leader changed from 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194) to 607da182484e4982a5d88b345ded418c (127.29.21.193). New cstate: current_term: 4 leader_uuid: "607da182484e4982a5d88b345ded418c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: UNKNOWN } } }
I20260805 22:22:57.164449 29994 raft_consensus.cc:2454] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 4.
I20260805 22:22:57.165669 30169 leader_election.cc:302] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09c9abba79e64e92b71def50c1afbe27, 607da182484e4982a5d88b345ded418c; no voters:
I20260805 22:22:57.165966 30112 raft_consensus.cc:2454] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09c9abba79e64e92b71def50c1afbe27 in term 4.
I20260805 22:22:57.166832 30480 raft_consensus.cc:2781] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:22:57.167371 30480 raft_consensus.cc:686] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 4 LEADER]: Becoming Leader. State: Replica: 09c9abba79e64e92b71def50c1afbe27, State: Running, Role: LEADER
I20260805 22:22:57.168263 30480 consensus_queue.cc:237] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } }
I20260805 22:22:57.175747 29837 catalog_manager.cc:5495] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 reported cstate change: term changed from 3 to 4, leader changed from 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194) to 09c9abba79e64e92b71def50c1afbe27 (127.29.21.195). New cstate: current_term: 4 leader_uuid: "09c9abba79e64e92b71def50c1afbe27" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: UNKNOWN } } }
I20260805 22:22:57.177106 29835 catalog_manager.cc:5495] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c reported cstate change: term changed from 2 to 3, leader changed from 1daa858569c149c5a44f92fe8c9bc635 (127.29.21.194) to 607da182484e4982a5d88b345ded418c (127.29.21.193). New cstate: current_term: 3 leader_uuid: "607da182484e4982a5d88b345ded418c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 } health_report { overall_health: UNKNOWN } } }
I20260805 22:22:57.405968 29994 raft_consensus.cc:1261] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 4 FOLLOWER]: Refusing update from remote peer 09c9abba79e64e92b71def50c1afbe27: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:22:57.407223 30480 consensus_queue.cc:1035] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:22:57.419432 30112 raft_consensus.cc:1261] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 4 FOLLOWER]: Refusing update from remote peer 09c9abba79e64e92b71def50c1afbe27: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:22:57.420733 30480 consensus_queue.cc:1035] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:22:57.470546 30112 raft_consensus.cc:1261] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Refusing update from remote peer 607da182484e4982a5d88b345ded418c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:22:57.471786 30511 consensus_queue.cc:1035] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:22:57.480206 30213 raft_consensus.cc:1261] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Refusing update from remote peer 607da182484e4982a5d88b345ded418c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:22:57.481757 30511 consensus_queue.cc:1035] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:22:57.568945 30213 raft_consensus.cc:1261] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 4 FOLLOWER]: Refusing update from remote peer 607da182484e4982a5d88b345ded418c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:22:57.570294 30511 consensus_queue.cc:1035] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:22:57.575179 29994 raft_consensus.cc:1261] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 4 FOLLOWER]: Refusing update from remote peer 09c9abba79e64e92b71def50c1afbe27: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:22:57.576691 30480 consensus_queue.cc:1035] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "607da182484e4982a5d88b345ded418c" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 39793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:22:57.578687 30116 raft_consensus.cc:1261] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 4 FOLLOWER]: Refusing update from remote peer 607da182484e4982a5d88b345ded418c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:22:57.580088 30524 consensus_queue.cc:1035] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:22:57.590472 30116 raft_consensus.cc:1261] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 4 FOLLOWER]: Refusing update from remote peer 09c9abba79e64e92b71def50c1afbe27: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:22:57.592181 30519 consensus_queue.cc:1035] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:22:57.641980 30116 raft_consensus.cc:1261] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Refusing update from remote peer 607da182484e4982a5d88b345ded418c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:22:57.643177 30509 consensus_queue.cc:1035] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1daa858569c149c5a44f92fe8c9bc635" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 39069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:22:57.651420 30213 raft_consensus.cc:1261] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Refusing update from remote peer 607da182484e4982a5d88b345ded418c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:22:57.652874 30537 consensus_queue.cc:1035] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [LEADER]: Connected to new peer: Peer: permanent_uuid: "09c9abba79e64e92b71def50c1afbe27" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 39287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:22:57.939641 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:22:57.945612 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:22:57.946003 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:22:58.946797 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:22:58.951997 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:22:58.952421 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:22:59.953279 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:22:59.958294 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:22:59.958737 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:00.959613 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:00.964308 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:00.964730 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:02.694851 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:02.699707 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:02.700145 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260805 22:23:02.705082 30242 debug-util.cc:398] Leaking SignalData structure 0x7b0800173060 after lost signal to thread 29786
W20260805 22:23:02.706539 30242 debug-util.cc:398] Leaking SignalData structure 0x7b0800173740 after lost signal to thread 29898
W20260805 22:23:02.707930 30242 debug-util.cc:398] Leaking SignalData structure 0x7b0800131420 after lost signal to thread 30035
W20260805 22:23:02.709028 30242 debug-util.cc:398] Leaking SignalData structure 0x7b08002034a0 after lost signal to thread 30152
W20260805 22:23:02.710099 30242 debug-util.cc:398] Leaking SignalData structure 0x7b08002027e0 after lost signal to thread 30247
I20260805 22:23:03.701213 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:03.708850 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:03.709415 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:04.710430 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:04.717677 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:04.718179 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:05.718966 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:05.724117 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:05.724565 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:06.725967 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:06.731385 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:06.731773 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:07.758085 29783 auto_leader_rebalancer-test.cc:314] The leader distribution is
I20260805 22:23:07.758666 29783 tablet_server.cc:178] TabletServer@127.29.21.193:0 shutting down...
I20260805 22:23:07.785281 29783 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260805 22:23:07.785998 29783 tablet_replica.cc:330] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c: stopping tablet replica
I20260805 22:23:07.786782 29783 raft_consensus.cc:2229] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 4 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:07.787375 29783 raft_consensus.cc:2258] T 777755da0c344319aaa4f32ec55ce02a P 607da182484e4982a5d88b345ded418c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:07.789837 29783 tablet_replica.cc:330] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c: stopping tablet replica
I20260805 22:23:07.790458 29783 raft_consensus.cc:2229] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [term 3 LEADER]: Raft consensus shutting down.
I20260805 22:23:07.791307 29783 raft_consensus.cc:2258] T 05d7ac208d874bdab1e7ee5d1909814d P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:07.792939 29783 tablet_replica.cc:330] T a7d64cdb99414419b519f3abb289edb1 P 607da182484e4982a5d88b345ded418c: stopping tablet replica
I20260805 22:23:07.793408 29783 raft_consensus.cc:2229] T a7d64cdb99414419b519f3abb289edb1 P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:07.793776 29783 raft_consensus.cc:2258] T a7d64cdb99414419b519f3abb289edb1 P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:07.795826 29783 tablet_replica.cc:330] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c: stopping tablet replica
I20260805 22:23:07.796305 29783 raft_consensus.cc:2229] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 4 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:07.796845 29783 raft_consensus.cc:2258] T c9ff840de5194b14935947f0ee7ee529 P 607da182484e4982a5d88b345ded418c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:07.799162 29783 tablet_replica.cc:330] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c: stopping tablet replica
I20260805 22:23:07.799633 29783 raft_consensus.cc:2229] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:07.799984 29783 raft_consensus.cc:2258] T 6267e585b62945d492c1fbb3c737c38d P 607da182484e4982a5d88b345ded418c [term 2 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:07.801507 29783 tablet_replica.cc:330] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c: stopping tablet replica
I20260805 22:23:07.802029 29783 raft_consensus.cc:2229] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:07.802372 29783 raft_consensus.cc:2258] T a8f348c46b854206beab7393137dccb5 P 607da182484e4982a5d88b345ded418c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:07.803915 29783 tablet_replica.cc:330] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c: stopping tablet replica
I20260805 22:23:07.804322 29783 raft_consensus.cc:2229] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c [term 4 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:07.804670 29783 raft_consensus.cc:2258] T 3e86514dd0434e69ae26b33066fc8bc0 P 607da182484e4982a5d88b345ded418c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:07.806210 29783 tablet_replica.cc:330] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c: stopping tablet replica
I20260805 22:23:07.806635 29783 raft_consensus.cc:2229] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 4 LEADER]: Raft consensus shutting down.
I20260805 22:23:07.807381 29783 raft_consensus.cc:2258] T 2230cd3dbabd47788d44be34d3d596c2 P 607da182484e4982a5d88b345ded418c [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:07.809355 29783 tablet_replica.cc:330] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c: stopping tablet replica
I20260805 22:23:07.810047 29783 raft_consensus.cc:2229] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 3 LEADER]: Raft consensus shutting down.
W20260805 22:23:07.809970 30064 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer
I20260805 22:23:07.811261 29783 raft_consensus.cc:2258] T b676fd9c96e94b01b7e1a9b297d8221f P 607da182484e4982a5d88b345ded418c [term 3 FOLLOWER]: Raft consensus is shut down!
W20260805 22:23:07.813302 30064 consensus_peers.cc:487] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 -> Peer 607da182484e4982a5d88b345ded418c (127.29.21.193:39793): Couldn't send request to peer 607da182484e4982a5d88b345ded418c. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:07.925837 29783 tablet_server.cc:194] TabletServer@127.29.21.193:0 shutdown complete.
W20260805 22:23:07.926298 30169 consensus_peers.cc:487] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 -> Peer 607da182484e4982a5d88b345ded418c (127.29.21.193:39793): Couldn't send request to peer 607da182484e4982a5d88b345ded418c. Status: Network error: Client connection negotiation failed: client connection to 127.29.21.193:39793: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:07.945592 29783 tablet_server.cc:178] TabletServer@127.29.21.194:0 shutting down...
I20260805 22:23:07.974499 29783 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260805 22:23:07.975158 29783 tablet_replica.cc:330] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635: stopping tablet replica
I20260805 22:23:07.975728 29783 raft_consensus.cc:2229] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:23:07.976537 29783 raft_consensus.cc:2258] T a8f348c46b854206beab7393137dccb5 P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:07.978369 29783 tablet_replica.cc:330] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635: stopping tablet replica
I20260805 22:23:07.978855 29783 raft_consensus.cc:2229] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:07.979302 29783 raft_consensus.cc:2258] T 05d7ac208d874bdab1e7ee5d1909814d P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:07.980924 29783 tablet_replica.cc:330] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635: stopping tablet replica
I20260805 22:23:07.981439 29783 raft_consensus.cc:2229] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:07.981989 29783 raft_consensus.cc:2258] T b676fd9c96e94b01b7e1a9b297d8221f P 1daa858569c149c5a44f92fe8c9bc635 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:07.984025 29783 tablet_replica.cc:330] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635: stopping tablet replica
I20260805 22:23:07.984591 29783 raft_consensus.cc:2229] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:07.985110 29783 raft_consensus.cc:2258] T 2230cd3dbabd47788d44be34d3d596c2 P 1daa858569c149c5a44f92fe8c9bc635 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:07.986848 29783 tablet_replica.cc:330] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635: stopping tablet replica
I20260805 22:23:07.987365 29783 raft_consensus.cc:2229] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 4 LEADER]: Raft consensus shutting down.
W20260805 22:23:07.987484 30064 consensus_peers.cc:487] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 -> Peer 607da182484e4982a5d88b345ded418c (127.29.21.193:39793): Couldn't send request to peer 607da182484e4982a5d88b345ded418c. Status: Network error: Client connection negotiation failed: client connection to 127.29.21.193:39793: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:07.988332 29783 raft_consensus.cc:2258] T 3e86514dd0434e69ae26b33066fc8bc0 P 1daa858569c149c5a44f92fe8c9bc635 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:07.989934 29783 tablet_replica.cc:330] T a7d64cdb99414419b519f3abb289edb1 P 1daa858569c149c5a44f92fe8c9bc635: stopping tablet replica
I20260805 22:23:07.990394 29783 raft_consensus.cc:2229] T a7d64cdb99414419b519f3abb289edb1 P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:07.990725 29783 raft_consensus.cc:2258] T a7d64cdb99414419b519f3abb289edb1 P 1daa858569c149c5a44f92fe8c9bc635 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:07.992303 29783 tablet_replica.cc:330] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635: stopping tablet replica
I20260805 22:23:07.992697 29783 raft_consensus.cc:2229] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:07.993031 29783 raft_consensus.cc:2258] T c9ff840de5194b14935947f0ee7ee529 P 1daa858569c149c5a44f92fe8c9bc635 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:07.994498 29783 tablet_replica.cc:330] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635: stopping tablet replica
I20260805 22:23:07.994932 29783 raft_consensus.cc:2229] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [term 2 LEADER]: Raft consensus shutting down.
I20260805 22:23:07.995586 29783 raft_consensus.cc:2258] T 6267e585b62945d492c1fbb3c737c38d P 1daa858569c149c5a44f92fe8c9bc635 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:07.997215 29783 tablet_replica.cc:330] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635: stopping tablet replica
I20260805 22:23:07.997659 29783 raft_consensus.cc:2229] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:07.998003 29783 raft_consensus.cc:2258] T 777755da0c344319aaa4f32ec55ce02a P 1daa858569c149c5a44f92fe8c9bc635 [term 4 FOLLOWER]: Raft consensus is shut down!
W20260805 22:23:08.006664 30169 consensus_peers.cc:487] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 -> Peer 607da182484e4982a5d88b345ded418c (127.29.21.193:39793): Couldn't send request to peer 607da182484e4982a5d88b345ded418c. Status: Network error: Client connection negotiation failed: client connection to 127.29.21.193:39793: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:08.025946 29783 tablet_server.cc:194] TabletServer@127.29.21.194:0 shutdown complete.
I20260805 22:23:08.044929 29783 tablet_server.cc:178] TabletServer@127.29.21.195:0 shutting down...
W20260805 22:23:08.071352 30169 consensus_peers.cc:487] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 -> Peer 607da182484e4982a5d88b345ded418c (127.29.21.193:39793): Couldn't send request to peer 607da182484e4982a5d88b345ded418c. Status: Network error: Client connection negotiation failed: client connection to 127.29.21.193:39793: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:08.073977 29783 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260805 22:23:08.074637 29783 tablet_replica.cc:330] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27: stopping tablet replica
I20260805 22:23:08.075233 29783 raft_consensus.cc:2229] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:08.075724 29783 raft_consensus.cc:2258] T b676fd9c96e94b01b7e1a9b297d8221f P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:08.077585 29783 tablet_replica.cc:330] T a8f348c46b854206beab7393137dccb5 P 09c9abba79e64e92b71def50c1afbe27: stopping tablet replica
I20260805 22:23:08.078060 29783 raft_consensus.cc:2229] T a8f348c46b854206beab7393137dccb5 P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:08.078469 29783 raft_consensus.cc:2258] T a8f348c46b854206beab7393137dccb5 P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:08.080212 29783 tablet_replica.cc:330] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27: stopping tablet replica
I20260805 22:23:08.080776 29783 raft_consensus.cc:2229] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:08.081207 29783 raft_consensus.cc:2258] T 3e86514dd0434e69ae26b33066fc8bc0 P 09c9abba79e64e92b71def50c1afbe27 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:08.082980 29783 tablet_replica.cc:330] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27: stopping tablet replica
I20260805 22:23:08.083446 29783 raft_consensus.cc:2229] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:08.083897 29783 raft_consensus.cc:2258] T 2230cd3dbabd47788d44be34d3d596c2 P 09c9abba79e64e92b71def50c1afbe27 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:08.085575 29783 tablet_replica.cc:330] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27: stopping tablet replica
I20260805 22:23:08.086035 29783 raft_consensus.cc:2229] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:08.086424 29783 raft_consensus.cc:2258] T 6267e585b62945d492c1fbb3c737c38d P 09c9abba79e64e92b71def50c1afbe27 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:08.088097 29783 tablet_replica.cc:330] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27: stopping tablet replica
I20260805 22:23:08.088537 29783 raft_consensus.cc:2229] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:23:08.089299 29783 raft_consensus.cc:2258] T a7d64cdb99414419b519f3abb289edb1 P 09c9abba79e64e92b71def50c1afbe27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:08.091022 29783 tablet_replica.cc:330] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27: stopping tablet replica
I20260805 22:23:08.091485 29783 raft_consensus.cc:2229] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 4 LEADER]: Raft consensus shutting down.
I20260805 22:23:08.092252 29783 raft_consensus.cc:2258] T c9ff840de5194b14935947f0ee7ee529 P 09c9abba79e64e92b71def50c1afbe27 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:08.093984 29783 tablet_replica.cc:330] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27: stopping tablet replica
I20260805 22:23:08.094424 29783 raft_consensus.cc:2229] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:08.094821 29783 raft_consensus.cc:2258] T 05d7ac208d874bdab1e7ee5d1909814d P 09c9abba79e64e92b71def50c1afbe27 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:08.096433 29783 tablet_replica.cc:330] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27: stopping tablet replica
I20260805 22:23:08.096848 29783 raft_consensus.cc:2229] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 4 LEADER]: Raft consensus shutting down.
I20260805 22:23:08.097584 29783 raft_consensus.cc:2258] T 777755da0c344319aaa4f32ec55ce02a P 09c9abba79e64e92b71def50c1afbe27 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:08.141491 29783 tablet_server.cc:194] TabletServer@127.29.21.195:0 shutdown complete.
I20260805 22:23:08.159853 29783 master.cc:537] Master@127.29.21.254:40149 shutting down...
I20260805 22:23:08.312781 29783 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:23:08.313465 29783 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:08.313865 29783 tablet_replica.cc:330] T 00000000000000000000000000000000 P 7dba7f8683554be786e28403f0117dfb: stopping tablet replica
I20260805 22:23:08.331664 29783 master.cc:552] Master@127.29.21.254:40149 shutdown complete.
09c9abba79e64e92b71def50c1afbe27 1
1daa858569c149c5a44f92fe8c9bc635 4
607da182484e4982a5d88b345ded418c 4
09c9abba79e64e92b71def50c1afbe27 3
1daa858569c149c5a44f92fe8c9bc635 3
607da182484e4982a5d88b345ded418c 3
09c9abba79e64e92b71def50c1afbe27 1
1daa858569c149c5a44f92fe8c9bc635 8
607da182484e4982a5d88b345ded418c 0
09c9abba79e64e92b71def50c1afbe27 3
1daa858569c149c5a44f92fe8c9bc635 3
607da182484e4982a5d88b345ded418c 3
[ OK ] LeaderRebalancerTest.FunctionalTestForDivided (32691 ms)
[ RUN ] LeaderRebalancerTest.FunctionalTestForNotDivided
I20260805 22:23:08.372130 29783 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.21.254:39533
I20260805 22:23:08.373181 29783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:23:08.378806 30579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:23:08.378891 30581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:23:08.379891 30584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:23:08.380095 29783 server_base.cc:1034] running on GCE node
I20260805 22:23:08.381210 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260805 22:23:08.381461 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260805 22:23:08.381583 29783 hybrid_clock.cc:648] HybridClock initialized: now 1785968588381575 us; error 0 us; skew 500 ppm
I20260805 22:23:08.382094 29783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:23:08.388913 29783 webserver.cc:460] Webserver started at http://127.29.21.254:44885/ using document root <none> and password file <none>
I20260805 22:23:08.389822 29783 fs_manager.cc:359] Metadata directory not provided
I20260805 22:23:08.389990 29783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:23:08.390224 29783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:23:08.391341 29783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/master-0-root/instance:
uuid: "154e2d921d3d4e94b77b95c4a741385c"
format_stamp: "Formatted at 2026-08-05 22:23:08 on dist-test-slave-f0tn"
I20260805 22:23:08.395975 29783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260805 22:23:08.399220 30590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:08.399909 29783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260805 22:23:08.400163 29783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/master-0-root
uuid: "154e2d921d3d4e94b77b95c4a741385c"
format_stamp: "Formatted at 2026-08-05 22:23:08 on dist-test-slave-f0tn"
I20260805 22:23:08.400413 29783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:23:08.420490 29783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:23:08.421819 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:23:08.458806 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.254:39533
I20260805 22:23:08.458920 30668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.21.254:39533 every 8 connection(s)
I20260805 22:23:08.462633 30669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:08.471936 30669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c: Bootstrap starting.
I20260805 22:23:08.476349 30669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:08.480783 30669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c: No bootstrap required, opened a new log
I20260805 22:23:08.482951 30669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154e2d921d3d4e94b77b95c4a741385c" member_type: VOTER }
I20260805 22:23:08.483448 30669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:08.483685 30669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 154e2d921d3d4e94b77b95c4a741385c, State: Initialized, Role: FOLLOWER
I20260805 22:23:08.484273 30669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154e2d921d3d4e94b77b95c4a741385c" member_type: VOTER }
I20260805 22:23:08.484737 30669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:23:08.484982 30669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:23:08.485248 30669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:08.489153 30669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154e2d921d3d4e94b77b95c4a741385c" member_type: VOTER }
I20260805 22:23:08.489679 30669 leader_election.cc:302] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 154e2d921d3d4e94b77b95c4a741385c; no voters:
I20260805 22:23:08.490792 30669 leader_election.cc:288] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:23:08.491205 30672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:08.492511 30672 raft_consensus.cc:686] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [term 1 LEADER]: Becoming Leader. State: Replica: 154e2d921d3d4e94b77b95c4a741385c, State: Running, Role: LEADER
I20260805 22:23:08.493103 30672 consensus_queue.cc:237] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154e2d921d3d4e94b77b95c4a741385c" member_type: VOTER }
I20260805 22:23:08.493733 30669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:23:08.499850 30673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "154e2d921d3d4e94b77b95c4a741385c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154e2d921d3d4e94b77b95c4a741385c" member_type: VOTER } }
I20260805 22:23:08.499958 30674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 154e2d921d3d4e94b77b95c4a741385c. Latest consensus state: current_term: 1 leader_uuid: "154e2d921d3d4e94b77b95c4a741385c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "154e2d921d3d4e94b77b95c4a741385c" member_type: VOTER } }
I20260805 22:23:08.500341 30673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [sys.catalog]: This master's current role is: LEADER
I20260805 22:23:08.500675 30674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [sys.catalog]: This master's current role is: LEADER
I20260805 22:23:08.502813 30684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:23:08.510076 30684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:23:08.511050 29783 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260805 22:23:08.520287 30684 catalog_manager.cc:1347] Generated new cluster ID: 7686892241584f549118db11f9c27882
I20260805 22:23:08.520588 30684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:23:08.536630 30684 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:23:08.538429 30684 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:23:08.556905 30684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c: Generated new TSK 0
I20260805 22:23:08.557636 30684 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:23:08.577507 29783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:23:08.583604 30699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:23:08.584867 30700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:23:08.585737 30702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:23:08.586299 29783 server_base.cc:1034] running on GCE node
I20260805 22:23:08.587255 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260805 22:23:08.587450 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260805 22:23:08.587601 29783 hybrid_clock.cc:648] HybridClock initialized: now 1785968588587585 us; error 0 us; skew 500 ppm
I20260805 22:23:08.588176 29783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:23:08.590718 29783 webserver.cc:460] Webserver started at http://127.29.21.193:37187/ using document root <none> and password file <none>
I20260805 22:23:08.591236 29783 fs_manager.cc:359] Metadata directory not provided
I20260805 22:23:08.591421 29783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:23:08.591668 29783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:23:08.592860 29783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/ts-0-root/instance:
uuid: "9f55007ac96c440da32b0ef9e5e29281"
format_stamp: "Formatted at 2026-08-05 22:23:08 on dist-test-slave-f0tn"
I20260805 22:23:08.597796 29783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260805 22:23:08.601015 30708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:08.601853 29783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260805 22:23:08.602192 29783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/ts-0-root
uuid: "9f55007ac96c440da32b0ef9e5e29281"
format_stamp: "Formatted at 2026-08-05 22:23:08 on dist-test-slave-f0tn"
I20260805 22:23:08.602494 29783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:23:08.617509 29783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:23:08.618747 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:23:08.620193 29783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:23:08.622613 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:23:08.622813 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:08.623034 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:23:08.623191 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:08.661504 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.193:37435
I20260805 22:23:08.661613 30797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.21.193:37435 every 8 connection(s)
I20260805 22:23:08.665882 29783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:23:08.673048 30803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:23:08.674734 30808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:23:08.676095 30810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:23:08.677166 29783 server_base.cc:1034] running on GCE node
I20260805 22:23:08.679051 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260805 22:23:08.679229 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260805 22:23:08.679405 29783 hybrid_clock.cc:648] HybridClock initialized: now 1785968588679389 us; error 0 us; skew 500 ppm
I20260805 22:23:08.680009 29783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:23:08.680008 30799 heartbeater.cc:347] Connected to a master server at 127.29.21.254:39533
I20260805 22:23:08.680500 30799 heartbeater.cc:464] Registering TS with master...
I20260805 22:23:08.681278 30799 heartbeater.cc:511] Master 127.29.21.254:39533 requested a full tablet report, sending...
I20260805 22:23:08.682830 29783 webserver.cc:460] Webserver started at http://127.29.21.194:37477/ using document root <none> and password file <none>
I20260805 22:23:08.683326 29783 fs_manager.cc:359] Metadata directory not provided
I20260805 22:23:08.683357 30612 ts_manager.cc:194] Registered new tserver with Master: 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435)
I20260805 22:23:08.683612 29783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:23:08.683986 29783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:23:08.685261 29783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/ts-1-root/instance:
uuid: "e0691006952a44cfa5f2397f172bfcf3"
format_stamp: "Formatted at 2026-08-05 22:23:08 on dist-test-slave-f0tn"
I20260805 22:23:08.685632 30612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42008
I20260805 22:23:08.689762 29783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260805 22:23:08.693156 30815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:08.693956 29783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 22:23:08.694242 29783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/ts-1-root
uuid: "e0691006952a44cfa5f2397f172bfcf3"
format_stamp: "Formatted at 2026-08-05 22:23:08 on dist-test-slave-f0tn"
I20260805 22:23:08.694568 29783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:23:08.704290 29783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:23:08.705561 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:23:08.707000 29783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:23:08.709224 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:23:08.709452 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260805 22:23:08.709683 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:23:08.709829 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:08.749055 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.194:37397
I20260805 22:23:08.749158 30892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.21.194:37397 every 8 connection(s)
I20260805 22:23:08.753616 29783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:23:08.761710 30898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:23:08.761710 30900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:23:08.764470 30903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:23:08.766067 29783 server_base.cc:1034] running on GCE node
I20260805 22:23:08.767083 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260805 22:23:08.767287 30894 heartbeater.cc:347] Connected to a master server at 127.29.21.254:39533
W20260805 22:23:08.767340 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260805 22:23:08.767684 29783 hybrid_clock.cc:648] HybridClock initialized: now 1785968588767665 us; error 0 us; skew 500 ppm
I20260805 22:23:08.767731 30894 heartbeater.cc:464] Registering TS with master...
I20260805 22:23:08.768270 29783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:23:08.768431 30894 heartbeater.cc:511] Master 127.29.21.254:39533 requested a full tablet report, sending...
I20260805 22:23:08.770418 30612 ts_manager.cc:194] Registered new tserver with Master: e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397)
I20260805 22:23:08.771142 29783 webserver.cc:460] Webserver started at http://127.29.21.195:36621/ using document root <none> and password file <none>
I20260805 22:23:08.771627 29783 fs_manager.cc:359] Metadata directory not provided
I20260805 22:23:08.771797 29783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:23:08.772025 29783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:23:08.772037 30612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42014
I20260805 22:23:08.773401 29783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/ts-2-root/instance:
uuid: "5366b554194f4acdb8916d0b1b2289e4"
format_stamp: "Formatted at 2026-08-05 22:23:08 on dist-test-slave-f0tn"
I20260805 22:23:08.777665 29783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260805 22:23:08.780601 30908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:08.781284 29783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260805 22:23:08.781589 29783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/ts-2-root
uuid: "5366b554194f4acdb8916d0b1b2289e4"
format_stamp: "Formatted at 2026-08-05 22:23:08 on dist-test-slave-f0tn"
I20260805 22:23:08.781877 29783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.FunctionalTestForNotDivided.1785968555326349-29783-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:23:08.802903 29783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:23:08.804155 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:23:08.805652 29783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:23:08.807852 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:23:08.808037 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:08.808252 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:23:08.808393 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:08.846697 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.195:33339
I20260805 22:23:08.846772 30993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.21.195:33339 every 8 connection(s)
I20260805 22:23:08.860857 30995 heartbeater.cc:347] Connected to a master server at 127.29.21.254:39533
I20260805 22:23:08.861235 30995 heartbeater.cc:464] Registering TS with master...
I20260805 22:23:08.861949 30995 heartbeater.cc:511] Master 127.29.21.254:39533 requested a full tablet report, sending...
I20260805 22:23:08.863691 30612 ts_manager.cc:194] Registered new tserver with Master: 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:08.864434 29783 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.014796259s
I20260805 22:23:08.865579 30612 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:42018
I20260805 22:23:08.867427 29783 test_util.cc:268] Using random seed: 928106319
I20260805 22:23:08.890481 30612 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42032:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s""\004\001\000\314\314\314\014\004\001\000\230\231\231\031\004\001\000dff&\004\001\0000333\004\001\000\374\377\377?\004\001\000\310\314\314L\004\001\000\224\231\231Y\004\001\000`fff\004\001\000,33s"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:23:08.893158 30612 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:23:08.967187 30848 tablet_service.cc:1467] Processing CreateTablet for tablet bd2a35893d484eae80fe9e16ba237fcc (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260805 22:23:08.968647 30848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd2a35893d484eae80fe9e16ba237fcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:08.969738 30847 tablet_service.cc:1467] Processing CreateTablet for tablet eaa4b450242b446fb1884b71a6bf55e5 (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260805 22:23:08.971089 30847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa4b450242b446fb1884b71a6bf55e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:08.971474 30846 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6d6d5cabf34c998bba078b482c0b05 (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260805 22:23:08.972877 30846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6d6d5cabf34c998bba078b482c0b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:08.973063 30948 tablet_service.cc:1467] Processing CreateTablet for tablet eaa4b450242b446fb1884b71a6bf55e5 (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260805 22:23:08.974316 30948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa4b450242b446fb1884b71a6bf55e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:08.975404 30842 tablet_service.cc:1467] Processing CreateTablet for tablet a4407f4a7a884d60aeae1a2bd23e76d5 (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260805 22:23:08.975939 30949 tablet_service.cc:1467] Processing CreateTablet for tablet bd2a35893d484eae80fe9e16ba237fcc (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260805 22:23:08.976666 30842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4407f4a7a884d60aeae1a2bd23e76d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:08.977160 30949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd2a35893d484eae80fe9e16ba237fcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:08.978567 30841 tablet_service.cc:1467] Processing CreateTablet for tablet de85cd15b9354ebbbd8be638aa855495 (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260805 22:23:08.979869 30841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de85cd15b9354ebbbd8be638aa855495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:08.979715 30946 tablet_service.cc:1467] Processing CreateTablet for tablet 7960edb18a3a4203b8c683499c8fe92c (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260805 22:23:08.980952 30946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7960edb18a3a4203b8c683499c8fe92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:08.982615 30944 tablet_service.cc:1467] Processing CreateTablet for tablet 5bbd9c17620b4dc1bfc9cd69188253e9 (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260805 22:23:08.983108 30840 tablet_service.cc:1467] Processing CreateTablet for tablet e9a7e5cf5aec44cf9cec9a2024024c5f (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260805 22:23:08.983916 30944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bbd9c17620b4dc1bfc9cd69188253e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:08.984395 30840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a7e5cf5aec44cf9cec9a2024024c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:08.986505 30943 tablet_service.cc:1467] Processing CreateTablet for tablet 77a3f906013c4bf3bc7b79a0777c0213 (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260805 22:23:08.987102 30839 tablet_service.cc:1467] Processing CreateTablet for tablet 8a62b2e2dc4145aca8e0a45586dfe57d (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260805 22:23:08.987809 30943 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a3f906013c4bf3bc7b79a0777c0213. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:08.988287 30839 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a62b2e2dc4145aca8e0a45586dfe57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:08.989887 30753 tablet_service.cc:1467] Processing CreateTablet for tablet bd2a35893d484eae80fe9e16ba237fcc (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION VALUES < 214748364
I20260805 22:23:08.991074 30753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd2a35893d484eae80fe9e16ba237fcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:08.992748 30742 tablet_service.cc:1467] Processing CreateTablet for tablet de85cd15b9354ebbbd8be638aa855495 (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260805 22:23:08.994107 30742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de85cd15b9354ebbbd8be638aa855495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:08.994505 30942 tablet_service.cc:1467] Processing CreateTablet for tablet a4407f4a7a884d60aeae1a2bd23e76d5 (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260805 22:23:08.996603 30941 tablet_service.cc:1467] Processing CreateTablet for tablet de85cd15b9354ebbbd8be638aa855495 (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 1503238548 <= VALUES < 1717986912
I20260805 22:23:08.997874 30941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de85cd15b9354ebbbd8be638aa855495. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:09.003193 30748 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6d6d5cabf34c998bba078b482c0b05 (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260805 22:23:09.003430 30940 tablet_service.cc:1467] Processing CreateTablet for tablet e9a7e5cf5aec44cf9cec9a2024024c5f (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260805 22:23:09.004555 30748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6d6d5cabf34c998bba078b482c0b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:09.004941 30940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a7e5cf5aec44cf9cec9a2024024c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:09.006585 30938 tablet_service.cc:1467] Processing CreateTablet for tablet 8a62b2e2dc4145aca8e0a45586dfe57d (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260805 22:23:09.008270 30749 tablet_service.cc:1467] Processing CreateTablet for tablet eaa4b450242b446fb1884b71a6bf55e5 (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 214748364 <= VALUES < 429496728
I20260805 22:23:09.009557 30749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eaa4b450242b446fb1884b71a6bf55e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:09.013625 30738 tablet_service.cc:1467] Processing CreateTablet for tablet e9a7e5cf5aec44cf9cec9a2024024c5f (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 1717986912 <= VALUES < 1932735276
I20260805 22:23:09.014525 30746 tablet_service.cc:1467] Processing CreateTablet for tablet 5bbd9c17620b4dc1bfc9cd69188253e9 (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260805 22:23:09.015125 30738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9a7e5cf5aec44cf9cec9a2024024c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:09.016250 30747 tablet_service.cc:1467] Processing CreateTablet for tablet 7960edb18a3a4203b8c683499c8fe92c (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260805 22:23:09.007050 30942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4407f4a7a884d60aeae1a2bd23e76d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:09.017570 30747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7960edb18a3a4203b8c683499c8fe92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:09.020136 30844 tablet_service.cc:1467] Processing CreateTablet for tablet 5bbd9c17620b4dc1bfc9cd69188253e9 (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 858993456 <= VALUES < 1073741820
I20260805 22:23:09.017671 30843 tablet_service.cc:1467] Processing CreateTablet for tablet 77a3f906013c4bf3bc7b79a0777c0213 (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260805 22:23:09.024596 30745 tablet_service.cc:1467] Processing CreateTablet for tablet 77a3f906013c4bf3bc7b79a0777c0213 (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 1073741820 <= VALUES < 1288490184
I20260805 22:23:09.032202 30845 tablet_service.cc:1467] Processing CreateTablet for tablet 7960edb18a3a4203b8c683499c8fe92c (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 644245092 <= VALUES < 858993456
I20260805 22:23:08.978569 30947 tablet_service.cc:1467] Processing CreateTablet for tablet 0a6d6d5cabf34c998bba078b482c0b05 (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 429496728 <= VALUES < 644245092
I20260805 22:23:09.035156 30947 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a6d6d5cabf34c998bba078b482c0b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:09.006501 30743 tablet_service.cc:1467] Processing CreateTablet for tablet a4407f4a7a884d60aeae1a2bd23e76d5 (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 1288490184 <= VALUES < 1503238548
I20260805 22:23:09.039116 30844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bbd9c17620b4dc1bfc9cd69188253e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:09.042235 30843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a3f906013c4bf3bc7b79a0777c0213. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:09.046106 30845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7960edb18a3a4203b8c683499c8fe92c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:09.053719 30938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a62b2e2dc4145aca8e0a45586dfe57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:09.064152 30746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bbd9c17620b4dc1bfc9cd69188253e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:09.067946 30737 tablet_service.cc:1467] Processing CreateTablet for tablet 8a62b2e2dc4145aca8e0a45586dfe57d (DEFAULT_TABLE table=test-workload [id=901a57ae4a2e4575808eec734058771b]), partition=RANGE (key) PARTITION 1932735276 <= VALUES
I20260805 22:23:09.068374 30745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77a3f906013c4bf3bc7b79a0777c0213. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:09.069346 30737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a62b2e2dc4145aca8e0a45586dfe57d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:09.069968 30743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4407f4a7a884d60aeae1a2bd23e76d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:09.153398 31014 tablet_bootstrap.cc:492] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4: Bootstrap starting.
I20260805 22:23:09.174233 31014 tablet_bootstrap.cc:654] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.185698 31017 tablet_bootstrap.cc:492] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3: Bootstrap starting.
I20260805 22:23:09.191365 31017 tablet_bootstrap.cc:654] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.196375 31016 tablet_bootstrap.cc:492] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281: Bootstrap starting.
I20260805 22:23:09.214401 31014 tablet_bootstrap.cc:492] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4: No bootstrap required, opened a new log
I20260805 22:23:09.214915 31014 ts_tablet_manager.cc:1397] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4: Time spent bootstrapping tablet: real 0.062s user 0.015s sys 0.013s
I20260805 22:23:09.217631 31014 raft_consensus.cc:348] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } }
I20260805 22:23:09.218367 31014 raft_consensus.cc:374] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.218654 31014 raft_consensus.cc:729] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.218987 31017 tablet_bootstrap.cc:492] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3: No bootstrap required, opened a new log
I20260805 22:23:09.219414 31017 ts_tablet_manager.cc:1397] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.003s
I20260805 22:23:09.219343 31014 consensus_queue.cc:260] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } }
I20260805 22:23:09.224505 31016 tablet_bootstrap.cc:654] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.227471 30995 heartbeater.cc:503] Master 127.29.21.254:39533 was elected leader, sending a full tablet report...
I20260805 22:23:09.227823 31014 ts_tablet_manager.cc:1428] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260805 22:23:09.227888 31017 raft_consensus.cc:348] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.228686 31017 raft_consensus.cc:374] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.228834 31014 tablet_bootstrap.cc:492] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4: Bootstrap starting.
I20260805 22:23:09.229000 31017 raft_consensus.cc:729] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.229816 31017 consensus_queue.cc:260] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.234982 31014 tablet_bootstrap.cc:654] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.235932 31017 ts_tablet_manager.cc:1428] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3: Time spent starting tablet: real 0.016s user 0.006s sys 0.004s
I20260805 22:23:09.236735 30894 heartbeater.cc:503] Master 127.29.21.254:39533 was elected leader, sending a full tablet report...
I20260805 22:23:09.245522 31017 tablet_bootstrap.cc:492] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3: Bootstrap starting.
I20260805 22:23:09.251863 31017 tablet_bootstrap.cc:654] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.258038 31014 tablet_bootstrap.cc:492] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4: No bootstrap required, opened a new log
I20260805 22:23:09.258620 31014 ts_tablet_manager.cc:1397] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4: Time spent bootstrapping tablet: real 0.030s user 0.018s sys 0.009s
I20260805 22:23:09.261911 31017 tablet_bootstrap.cc:492] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3: No bootstrap required, opened a new log
I20260805 22:23:09.262420 31017 ts_tablet_manager.cc:1397] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260805 22:23:09.262537 31014 raft_consensus.cc:348] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.263303 31014 raft_consensus.cc:374] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.263630 31014 raft_consensus.cc:729] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.264437 31014 consensus_queue.cc:260] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.265215 31017 raft_consensus.cc:348] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.265965 31017 raft_consensus.cc:374] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.266253 31017 raft_consensus.cc:729] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.267042 31017 consensus_queue.cc:260] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.268307 31014 ts_tablet_manager.cc:1428] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4: Time spent starting tablet: real 0.009s user 0.005s sys 0.002s
I20260805 22:23:09.269229 31014 tablet_bootstrap.cc:492] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4: Bootstrap starting.
I20260805 22:23:09.269570 31017 ts_tablet_manager.cc:1428] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260805 22:23:09.270408 31017 tablet_bootstrap.cc:492] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3: Bootstrap starting.
I20260805 22:23:09.272607 31016 tablet_bootstrap.cc:492] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281: No bootstrap required, opened a new log
I20260805 22:23:09.273034 31016 ts_tablet_manager.cc:1397] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281: Time spent bootstrapping tablet: real 0.077s user 0.026s sys 0.025s
I20260805 22:23:09.275086 31014 tablet_bootstrap.cc:654] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.276034 31017 tablet_bootstrap.cc:654] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.275628 31016 raft_consensus.cc:348] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } }
I20260805 22:23:09.276782 31016 raft_consensus.cc:374] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.277091 31016 raft_consensus.cc:729] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.277873 31016 consensus_queue.cc:260] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } }
I20260805 22:23:09.290117 31017 tablet_bootstrap.cc:492] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3: No bootstrap required, opened a new log
I20260805 22:23:09.290423 31014 tablet_bootstrap.cc:492] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4: No bootstrap required, opened a new log
I20260805 22:23:09.290382 30799 heartbeater.cc:503] Master 127.29.21.254:39533 was elected leader, sending a full tablet report...
I20260805 22:23:09.290720 31017 ts_tablet_manager.cc:1397] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.003s
I20260805 22:23:09.290822 31016 ts_tablet_manager.cc:1428] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
I20260805 22:23:09.291955 31016 tablet_bootstrap.cc:492] T 8a62b2e2dc4145aca8e0a45586dfe57d P 9f55007ac96c440da32b0ef9e5e29281: Bootstrap starting.
I20260805 22:23:09.293349 31014 ts_tablet_manager.cc:1397] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.007s
I20260805 22:23:09.293596 31017 raft_consensus.cc:348] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.294411 31017 raft_consensus.cc:374] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.294848 31017 raft_consensus.cc:729] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.295691 31017 consensus_queue.cc:260] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.296211 31014 raft_consensus.cc:348] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.297019 31014 raft_consensus.cc:374] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.297290 31014 raft_consensus.cc:729] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.297679 31016 tablet_bootstrap.cc:654] T 8a62b2e2dc4145aca8e0a45586dfe57d P 9f55007ac96c440da32b0ef9e5e29281: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.297973 31017 ts_tablet_manager.cc:1428] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260805 22:23:09.298065 31014 consensus_queue.cc:260] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.298908 31017 tablet_bootstrap.cc:492] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3: Bootstrap starting.
I20260805 22:23:09.301422 31014 ts_tablet_manager.cc:1428] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260805 22:23:09.302536 31014 tablet_bootstrap.cc:492] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4: Bootstrap starting.
I20260805 22:23:09.306529 31017 tablet_bootstrap.cc:654] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.308411 31014 tablet_bootstrap.cc:654] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.311975 31016 tablet_bootstrap.cc:492] T 8a62b2e2dc4145aca8e0a45586dfe57d P 9f55007ac96c440da32b0ef9e5e29281: No bootstrap required, opened a new log
I20260805 22:23:09.312496 31016 ts_tablet_manager.cc:1397] T 8a62b2e2dc4145aca8e0a45586dfe57d P 9f55007ac96c440da32b0ef9e5e29281: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.009s
I20260805 22:23:09.315232 31016 raft_consensus.cc:348] T 8a62b2e2dc4145aca8e0a45586dfe57d P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.315995 31016 raft_consensus.cc:374] T 8a62b2e2dc4145aca8e0a45586dfe57d P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.316298 31016 raft_consensus.cc:729] T 8a62b2e2dc4145aca8e0a45586dfe57d P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.317111 31016 consensus_queue.cc:260] T 8a62b2e2dc4145aca8e0a45586dfe57d P 9f55007ac96c440da32b0ef9e5e29281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.318212 31017 tablet_bootstrap.cc:492] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3: No bootstrap required, opened a new log
I20260805 22:23:09.318681 31017 ts_tablet_manager.cc:1397] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.007s
I20260805 22:23:09.319437 31016 ts_tablet_manager.cc:1428] T 8a62b2e2dc4145aca8e0a45586dfe57d P 9f55007ac96c440da32b0ef9e5e29281: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260805 22:23:09.320339 31016 tablet_bootstrap.cc:492] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281: Bootstrap starting.
I20260805 22:23:09.321187 31017 raft_consensus.cc:348] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } }
I20260805 22:23:09.322005 31017 raft_consensus.cc:374] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.322286 31017 raft_consensus.cc:729] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.323135 31017 consensus_queue.cc:260] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } }
I20260805 22:23:09.326707 31016 tablet_bootstrap.cc:654] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.328224 31017 ts_tablet_manager.cc:1428] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260805 22:23:09.329237 31017 tablet_bootstrap.cc:492] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3: Bootstrap starting.
I20260805 22:23:09.335402 31017 tablet_bootstrap.cc:654] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.338294 31016 tablet_bootstrap.cc:492] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281: No bootstrap required, opened a new log
I20260805 22:23:09.338769 31016 ts_tablet_manager.cc:1397] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20260805 22:23:09.339995 31014 tablet_bootstrap.cc:492] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4: No bootstrap required, opened a new log
I20260805 22:23:09.340466 31014 ts_tablet_manager.cc:1397] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.022s
I20260805 22:23:09.341588 31016 raft_consensus.cc:348] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.342456 31016 raft_consensus.cc:374] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.343034 31016 raft_consensus.cc:729] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.343083 31014 raft_consensus.cc:348] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.346189 31014 raft_consensus.cc:374] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.346508 31014 raft_consensus.cc:729] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.346333 31016 consensus_queue.cc:260] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.347316 31014 consensus_queue.cc:260] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.348860 31016 ts_tablet_manager.cc:1428] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260805 22:23:09.349818 31016 tablet_bootstrap.cc:492] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281: Bootstrap starting.
I20260805 22:23:09.351732 31014 ts_tablet_manager.cc:1428] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4: Time spent starting tablet: real 0.011s user 0.001s sys 0.008s
W20260805 22:23:09.351940 30996 tablet.cc:2369] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:23:09.352654 31014 tablet_bootstrap.cc:492] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4: Bootstrap starting.
I20260805 22:23:09.358631 31014 tablet_bootstrap.cc:654] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.360886 31016 tablet_bootstrap.cc:654] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.361167 31017 tablet_bootstrap.cc:492] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3: No bootstrap required, opened a new log
I20260805 22:23:09.361685 31017 ts_tablet_manager.cc:1397] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.009s
I20260805 22:23:09.364317 31017 raft_consensus.cc:348] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.365068 31017 raft_consensus.cc:374] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.365348 31017 raft_consensus.cc:729] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.366066 31017 consensus_queue.cc:260] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.370178 31017 ts_tablet_manager.cc:1428] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20260805 22:23:09.371084 31017 tablet_bootstrap.cc:492] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3: Bootstrap starting.
I20260805 22:23:09.371356 31016 tablet_bootstrap.cc:492] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281: No bootstrap required, opened a new log
I20260805 22:23:09.371501 31014 tablet_bootstrap.cc:492] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4: No bootstrap required, opened a new log
I20260805 22:23:09.371789 31016 ts_tablet_manager.cc:1397] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.003s
I20260805 22:23:09.372164 31014 ts_tablet_manager.cc:1397] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.005s
I20260805 22:23:09.374519 31016 raft_consensus.cc:348] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.375312 31016 raft_consensus.cc:374] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.374840 31014 raft_consensus.cc:348] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.375613 31016 raft_consensus.cc:729] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.375775 31014 raft_consensus.cc:374] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.376211 31014 raft_consensus.cc:729] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.376327 31016 consensus_queue.cc:260] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.377182 31017 tablet_bootstrap.cc:654] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.376981 31014 consensus_queue.cc:260] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.379688 31014 ts_tablet_manager.cc:1428] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260805 22:23:09.380672 31014 tablet_bootstrap.cc:492] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4: Bootstrap starting.
I20260805 22:23:09.381294 31016 ts_tablet_manager.cc:1428] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281: Time spent starting tablet: real 0.009s user 0.006s sys 0.001s
I20260805 22:23:09.382421 31016 tablet_bootstrap.cc:492] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281: Bootstrap starting.
I20260805 22:23:09.386545 31024 raft_consensus.cc:482] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:09.387012 31024 raft_consensus.cc:504] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.391510 31024 leader_election.cc:288] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:09.392019 31024 raft_consensus.cc:482] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:09.392827 31016 tablet_bootstrap.cc:654] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.393499 31014 tablet_bootstrap.cc:654] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.392447 31024 raft_consensus.cc:504] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.396566 31024 leader_election.cc:288] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:09.405251 31016 tablet_bootstrap.cc:492] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281: No bootstrap required, opened a new log
I20260805 22:23:09.405827 31016 ts_tablet_manager.cc:1397] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.009s
I20260805 22:23:09.407563 31014 tablet_bootstrap.cc:492] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4: No bootstrap required, opened a new log
I20260805 22:23:09.408025 31014 ts_tablet_manager.cc:1397] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.000s
I20260805 22:23:09.408537 31016 raft_consensus.cc:348] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.409276 31016 raft_consensus.cc:374] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.409601 31016 raft_consensus.cc:729] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.410436 31016 consensus_queue.cc:260] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.410665 31014 raft_consensus.cc:348] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.411403 31014 raft_consensus.cc:374] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.411700 31014 raft_consensus.cc:729] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.413007 31017 tablet_bootstrap.cc:492] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3: No bootstrap required, opened a new log
I20260805 22:23:09.412497 31014 consensus_queue.cc:260] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.413511 31017 ts_tablet_manager.cc:1397] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3: Time spent bootstrapping tablet: real 0.043s user 0.011s sys 0.009s
I20260805 22:23:09.414779 31028 raft_consensus.cc:482] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:09.415089 31014 ts_tablet_manager.cc:1428] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260805 22:23:09.415246 31028 raft_consensus.cc:504] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.416056 31014 tablet_bootstrap.cc:492] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4: Bootstrap starting.
I20260805 22:23:09.416265 31017 raft_consensus.cc:348] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.416991 31017 raft_consensus.cc:374] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.417511 31017 raft_consensus.cc:729] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.418198 31017 consensus_queue.cc:260] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.419976 31020 raft_consensus.cc:482] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:09.420513 31020 raft_consensus.cc:504] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.423171 31020 leader_election.cc:288] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397)
I20260805 22:23:09.425891 31014 tablet_bootstrap.cc:654] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.435043 31028 leader_election.cc:288] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:09.420854 31017 ts_tablet_manager.cc:1428] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260805 22:23:09.454352 31017 tablet_bootstrap.cc:492] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3: Bootstrap starting.
I20260805 22:23:09.458834 31016 ts_tablet_manager.cc:1428] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281: Time spent starting tablet: real 0.053s user 0.004s sys 0.002s
I20260805 22:23:09.459903 31016 tablet_bootstrap.cc:492] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281: Bootstrap starting.
I20260805 22:23:09.481209 31016 tablet_bootstrap.cc:654] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.485934 31028 raft_consensus.cc:482] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:09.486452 31028 raft_consensus.cc:504] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } }
I20260805 22:23:09.487317 30766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281" is_pre_election: true
I20260805 22:23:09.487901 31017 tablet_bootstrap.cc:654] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.488202 30766 raft_consensus.cc:2454] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 0.
I20260805 22:23:09.489171 31028 leader_election.cc:288] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:09.490343 31014 tablet_bootstrap.cc:492] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4: No bootstrap required, opened a new log
I20260805 22:23:09.491636 30820 leader_election.cc:302] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:09.491797 30766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a62b2e2dc4145aca8e0a45586dfe57d" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281" is_pre_election: true
I20260805 22:23:09.492925 31044 raft_consensus.cc:2781] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:09.490927 31014 ts_tablet_manager.cc:1397] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4: Time spent bootstrapping tablet: real 0.075s user 0.022s sys 0.025s
I20260805 22:23:09.493350 31044 raft_consensus.cc:482] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:09.493726 31044 raft_consensus.cc:3037] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.493013 30766 raft_consensus.cc:2454] T 8a62b2e2dc4145aca8e0a45586dfe57d P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 0.
I20260805 22:23:09.497534 31014 raft_consensus.cc:348] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.498003 30820 leader_election.cc:302] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:09.498354 31014 raft_consensus.cc:374] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.498690 31014 raft_consensus.cc:729] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.499413 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3" is_pre_election: true
I20260805 22:23:09.500602 30858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2a35893d484eae80fe9e16ba237fcc" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3" is_pre_election: true
I20260805 22:23:09.501080 31024 raft_consensus.cc:2781] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:09.501300 30858 raft_consensus.cc:2454] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 0.
I20260805 22:23:09.501439 31024 raft_consensus.cc:482] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:09.501824 31024 raft_consensus.cc:3037] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Advancing to term 1
W20260805 22:23:09.504570 30685 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260805 22:23:09.505910 30709 leader_election.cc:302] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:09.499415 31014 consensus_queue.cc:260] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.506822 31028 raft_consensus.cc:2781] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:09.507968 31028 raft_consensus.cc:482] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:09.508323 31028 raft_consensus.cc:3037] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.508836 31014 ts_tablet_manager.cc:1428] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260805 22:23:09.509864 31014 tablet_bootstrap.cc:492] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4: Bootstrap starting.
I20260805 22:23:09.510548 31017 tablet_bootstrap.cc:492] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3: No bootstrap required, opened a new log
I20260805 22:23:09.510993 31017 ts_tablet_manager.cc:1397] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3: Time spent bootstrapping tablet: real 0.057s user 0.010s sys 0.000s
I20260805 22:23:09.512814 30766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281" is_pre_election: true
I20260805 22:23:09.513455 30766 raft_consensus.cc:2454] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 0.
I20260805 22:23:09.513706 31017 raft_consensus.cc:348] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.514423 31017 raft_consensus.cc:374] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.514681 30916 leader_election.cc:302] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, 9f55007ac96c440da32b0ef9e5e29281; no voters:
I20260805 22:23:09.515622 31016 tablet_bootstrap.cc:492] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281: No bootstrap required, opened a new log
I20260805 22:23:09.516140 31016 ts_tablet_manager.cc:1397] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281: Time spent bootstrapping tablet: real 0.056s user 0.009s sys 0.014s
I20260805 22:23:09.516561 31014 tablet_bootstrap.cc:654] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.517927 31020 raft_consensus.cc:2781] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:09.518290 31020 raft_consensus.cc:482] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:09.518635 31020 raft_consensus.cc:3037] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.519402 31024 raft_consensus.cc:504] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.524399 31016 raft_consensus.cc:348] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.526407 31020 raft_consensus.cc:504] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.527405 31044 raft_consensus.cc:504] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.525144 31016 raft_consensus.cc:374] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.528470 31016 raft_consensus.cc:729] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.528618 30860 raft_consensus.cc:2379] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f55007ac96c440da32b0ef9e5e29281 in current term 1: Already voted for candidate e0691006952a44cfa5f2397f172bfcf3 in this term.
I20260805 22:23:09.530167 30766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a62b2e2dc4145aca8e0a45586dfe57d" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:09.530758 30766 raft_consensus.cc:3037] T 8a62b2e2dc4145aca8e0a45586dfe57d P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.530617 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4" is_pre_election: true
I20260805 22:23:09.531441 30963 raft_consensus.cc:2379] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f55007ac96c440da32b0ef9e5e29281 in current term 1: Already voted for candidate 5366b554194f4acdb8916d0b1b2289e4 in this term.
I20260805 22:23:09.531893 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:09.532580 30765 raft_consensus.cc:3037] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.529294 31016 consensus_queue.cc:260] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.537508 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4" is_pre_election: true
I20260805 22:23:09.538345 30963 raft_consensus.cc:2379] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e0691006952a44cfa5f2397f172bfcf3 in current term 1: Already voted for candidate 5366b554194f4acdb8916d0b1b2289e4 in this term.
I20260805 22:23:09.537536 30962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a62b2e2dc4145aca8e0a45586dfe57d" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4" is_pre_election: true
I20260805 22:23:09.539613 30962 raft_consensus.cc:2454] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 0.
I20260805 22:23:09.514714 31017 raft_consensus.cc:729] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.515282 31028 raft_consensus.cc:504] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } }
I20260805 22:23:09.543610 30962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2a35893d484eae80fe9e16ba237fcc" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4" is_pre_election: true
I20260805 22:23:09.544266 30962 raft_consensus.cc:2454] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 0.
I20260805 22:23:09.544160 31017 consensus_queue.cc:260] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.545215 31028 leader_election.cc:288] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 election: Requested vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:09.550153 30962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2a35893d484eae80fe9e16ba237fcc" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:09.550719 30962 raft_consensus.cc:3037] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.548496 31020 leader_election.cc:288] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397)
I20260805 22:23:09.553273 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a62b2e2dc4145aca8e0a45586dfe57d" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:09.554028 30963 raft_consensus.cc:3037] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.554582 30858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:09.555536 30858 raft_consensus.cc:2379] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5366b554194f4acdb8916d0b1b2289e4 in current term 1: Already voted for candidate e0691006952a44cfa5f2397f172bfcf3 in this term.
I20260805 22:23:09.557077 31028 raft_consensus.cc:482] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:09.558135 31028 raft_consensus.cc:504] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.556762 31044 leader_election.cc:288] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:09.558985 30962 raft_consensus.cc:2454] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 1.
I20260805 22:23:09.559767 30961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:09.560071 31028 leader_election.cc:288] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:09.560127 31020 raft_consensus.cc:482] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:09.560668 31020 raft_consensus.cc:504] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } }
I20260805 22:23:09.560966 30716 leader_election.cc:302] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, 9f55007ac96c440da32b0ef9e5e29281; no voters:
I20260805 22:23:09.561280 30962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbd9c17620b4dc1bfc9cd69188253e9" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4" is_pre_election: true
I20260805 22:23:09.562760 31044 raft_consensus.cc:482] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:09.564029 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3" is_pre_election: true
I20260805 22:23:09.564917 30860 raft_consensus.cc:2379] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5366b554194f4acdb8916d0b1b2289e4 in current term 1: Already voted for candidate e0691006952a44cfa5f2397f172bfcf3 in this term.
I20260805 22:23:09.562598 30764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:09.565303 30858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2a35893d484eae80fe9e16ba237fcc" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3" is_pre_election: true
I20260805 22:23:09.565982 30858 raft_consensus.cc:2454] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 0.
W20260805 22:23:09.563378 30716 leader_election.cc:341] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:09.564392 31044 raft_consensus.cc:504] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.567477 30961 raft_consensus.cc:2379] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e0691006952a44cfa5f2397f172bfcf3 in current term 1: Already voted for candidate 5366b554194f4acdb8916d0b1b2289e4 in this term.
I20260805 22:23:09.567855 30909 leader_election.cc:302] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:09.568892 30820 leader_election.cc:302] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0691006952a44cfa5f2397f172bfcf3; no voters: 5366b554194f4acdb8916d0b1b2289e4, 9f55007ac96c440da32b0ef9e5e29281
I20260805 22:23:09.571386 31020 leader_election.cc:288] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435)
I20260805 22:23:09.571826 31024 leader_election.cc:288] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:09.548681 30765 raft_consensus.cc:2454] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 1.
I20260805 22:23:09.573200 31044 leader_election.cc:288] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:09.573958 31028 raft_consensus.cc:2781] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:09.581878 30763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6d6d5cabf34c998bba078b482c0b05" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281" is_pre_election: true
W20260805 22:23:09.583112 30820 leader_election.cc:341] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:09.583734 31017 ts_tablet_manager.cc:1428] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3: Time spent starting tablet: real 0.072s user 0.010s sys 0.001s
I20260805 22:23:09.584522 31017 tablet_bootstrap.cc:492] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3: Bootstrap starting.
I20260805 22:23:09.586035 30961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6d6d5cabf34c998bba078b482c0b05" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4" is_pre_election: true
I20260805 22:23:09.585875 31016 ts_tablet_manager.cc:1428] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281: Time spent starting tablet: real 0.069s user 0.016s sys 0.003s
I20260805 22:23:09.577015 30764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2a35893d484eae80fe9e16ba237fcc" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281" is_pre_election: true
I20260805 22:23:09.579768 31020 raft_consensus.cc:482] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:09.580710 31024 raft_consensus.cc:2726] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260805 22:23:09.588886 30709 leader_election.cc:302] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281; no voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3
I20260805 22:23:09.589754 31020 raft_consensus.cc:504] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.590514 31016 tablet_bootstrap.cc:492] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281: Bootstrap starting.
I20260805 22:23:09.591012 31065 raft_consensus.cc:2726] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:09.591744 31020 leader_election.cc:288] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397)
I20260805 22:23:09.589129 30916 leader_election.cc:302] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, 9f55007ac96c440da32b0ef9e5e29281; no voters: e0691006952a44cfa5f2397f172bfcf3
I20260805 22:23:09.592648 30858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2a35893d484eae80fe9e16ba237fcc" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:09.593072 30766 raft_consensus.cc:2454] T 8a62b2e2dc4145aca8e0a45586dfe57d P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 1.
I20260805 22:23:09.593209 30858 raft_consensus.cc:3037] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.597939 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbd9c17620b4dc1bfc9cd69188253e9" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3" is_pre_election: true
W20260805 22:23:09.593499 30820 leader_election.cc:341] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:09.603061 30820 leader_election.cc:302] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0691006952a44cfa5f2397f172bfcf3; no voters: 5366b554194f4acdb8916d0b1b2289e4, 9f55007ac96c440da32b0ef9e5e29281
I20260805 22:23:09.603856 31024 raft_consensus.cc:2726] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260805 22:23:09.604476 30709 leader_election.cc:341] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:09.593799 31020 raft_consensus.cc:2781] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:09.589416 31028 raft_consensus.cc:686] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 1 LEADER]: Becoming Leader. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Running, Role: LEADER
I20260805 22:23:09.605481 31020 raft_consensus.cc:686] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 LEADER]: Becoming Leader. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Running, Role: LEADER
I20260805 22:23:09.605975 31028 consensus_queue.cc:237] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } }
I20260805 22:23:09.606441 31020 consensus_queue.cc:237] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.609045 31024 raft_consensus.cc:482] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:09.609525 30709 leader_election.cc:302] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281; no voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3
I20260805 22:23:09.609530 31024 raft_consensus.cc:504] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.596014 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281" is_pre_election: true
I20260805 22:23:09.601158 30963 raft_consensus.cc:2454] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 1.
I20260805 22:23:09.614038 30820 leader_election.cc:302] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281, e0691006952a44cfa5f2397f172bfcf3; no voters:
W20260805 22:23:09.615658 30916 leader_election.cc:341] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:09.616041 31068 raft_consensus.cc:2781] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:09.616648 31024 leader_election.cc:288] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:09.617179 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3" is_pre_election: true
I20260805 22:23:09.617717 30961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4" is_pre_election: true
I20260805 22:23:09.618074 30860 raft_consensus.cc:2454] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 0.
I20260805 22:23:09.618304 30961 raft_consensus.cc:2454] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 0.
I20260805 22:23:09.619287 30909 leader_election.cc:302] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3; no voters: 9f55007ac96c440da32b0ef9e5e29281
I20260805 22:23:09.620316 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281" is_pre_election: true
I20260805 22:23:09.616743 31068 raft_consensus.cc:686] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 1 LEADER]: Becoming Leader. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Running, Role: LEADER
I20260805 22:23:09.621034 31063 raft_consensus.cc:2781] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:23:09.621448 31063 raft_consensus.cc:482] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:09.621829 31063 raft_consensus.cc:3037] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:09.620561 31070 raft_consensus.cc:2781] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260805 22:23:09.622326 30820 leader_election.cc:341] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:09.620612 31017 tablet_bootstrap.cc:654] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.624441 30820 leader_election.cc:302] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3; no voters: 9f55007ac96c440da32b0ef9e5e29281
I20260805 22:23:09.624778 30858 raft_consensus.cc:2454] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 1.
I20260805 22:23:09.625344 31024 raft_consensus.cc:2781] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:09.625735 31024 raft_consensus.cc:482] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:09.626032 31024 raft_consensus.cc:3037] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.621906 31068 consensus_queue.cc:237] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.613940 31065 raft_consensus.cc:2726] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:09.628237 31070 raft_consensus.cc:482] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:09.629333 31070 raft_consensus.cc:3037] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.632365 31014 tablet_bootstrap.cc:492] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4: No bootstrap required, opened a new log
I20260805 22:23:09.632853 31014 ts_tablet_manager.cc:1397] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4: Time spent bootstrapping tablet: real 0.123s user 0.004s sys 0.046s
I20260805 22:23:09.648639 31067 raft_consensus.cc:482] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:09.650045 31016 tablet_bootstrap.cc:654] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.649447 31067 raft_consensus.cc:504] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.652630 31067 leader_election.cc:288] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:09.655510 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de85cd15b9354ebbbd8be638aa855495" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281" is_pre_election: true
I20260805 22:23:09.656419 31014 raft_consensus.cc:348] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
W20260805 22:23:09.656879 30820 leader_election.cc:341] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:09.656888 30961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de85cd15b9354ebbbd8be638aa855495" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4" is_pre_election: true
I20260805 22:23:09.657680 30961 raft_consensus.cc:2454] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 0.
I20260805 22:23:09.658772 30820 leader_election.cc:302] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3; no voters: 9f55007ac96c440da32b0ef9e5e29281
I20260805 22:23:09.659773 31067 raft_consensus.cc:2781] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:09.660135 31067 raft_consensus.cc:482] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:09.660074 31024 raft_consensus.cc:504] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.660583 31067 raft_consensus.cc:3037] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.662137 31024 leader_election.cc:288] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:09.661228 30609 catalog_manager.cc:5495] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "5366b554194f4acdb8916d0b1b2289e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:09.657106 31014 raft_consensus.cc:374] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.663250 31014 raft_consensus.cc:729] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.664338 31014 consensus_queue.cc:260] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.666365 31014 ts_tablet_manager.cc:1428] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4: Time spent starting tablet: real 0.033s user 0.007s sys 0.006s
I20260805 22:23:09.666219 30961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:09.667236 31014 tablet_bootstrap.cc:492] T 0a6d6d5cabf34c998bba078b482c0b05 P 5366b554194f4acdb8916d0b1b2289e4: Bootstrap starting.
I20260805 22:23:09.661902 31063 raft_consensus.cc:504] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } }
I20260805 22:23:09.666926 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:09.672350 31017 tablet_bootstrap.cc:492] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3: No bootstrap required, opened a new log
I20260805 22:23:09.675124 31014 tablet_bootstrap.cc:654] T 0a6d6d5cabf34c998bba078b482c0b05 P 5366b554194f4acdb8916d0b1b2289e4: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.672922 31020 raft_consensus.cc:482] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:09.684844 31017 ts_tablet_manager.cc:1397] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3: Time spent bootstrapping tablet: real 0.100s user 0.019s sys 0.011s
W20260805 22:23:09.674180 30820 leader_election.cc:341] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:09.686167 31020 raft_consensus.cc:504] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.689535 31020 leader_election.cc:288] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397)
I20260805 22:23:09.689347 31067 raft_consensus.cc:504] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.691736 31067 leader_election.cc:288] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:09.691869 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a7e5cf5aec44cf9cec9a2024024c5f" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3" is_pre_election: true
I20260805 22:23:09.692649 30860 raft_consensus.cc:2454] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 0.
I20260805 22:23:09.692816 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de85cd15b9354ebbbd8be638aa855495" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:09.689503 30858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2a35893d484eae80fe9e16ba237fcc" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:09.693959 30858 raft_consensus.cc:3037] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Advancing to term 2
W20260805 22:23:09.694242 30820 leader_election.cc:341] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:09.694435 30909 leader_election.cc:302] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:09.694898 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a7e5cf5aec44cf9cec9a2024024c5f" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281" is_pre_election: true
I20260805 22:23:09.695408 31020 raft_consensus.cc:2781] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:09.695808 31020 raft_consensus.cc:482] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:09.696130 31020 raft_consensus.cc:3037] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Advancing to term 1
W20260805 22:23:09.696354 30916 leader_election.cc:341] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:09.693106 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de85cd15b9354ebbbd8be638aa855495" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:09.697937 30963 raft_consensus.cc:3037] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.701189 30858 raft_consensus.cc:2454] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 2.
I20260805 22:23:09.702944 30909 leader_election.cc:302] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:09.705533 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2a35893d484eae80fe9e16ba237fcc" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:09.706307 31020 raft_consensus.cc:504] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.702876 31063 leader_election.cc:288] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 2 election: Requested vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435)
I20260805 22:23:09.707839 31063 raft_consensus.cc:2781] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:23:09.708353 31063 raft_consensus.cc:686] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 2 LEADER]: Becoming Leader. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Running, Role: LEADER
I20260805 22:23:09.709197 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a7e5cf5aec44cf9cec9a2024024c5f" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
W20260805 22:23:09.710673 30916 leader_election.cc:341] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:09.712488 31020 leader_election.cc:288] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397)
I20260805 22:23:09.717172 31017 raft_consensus.cc:348] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.718005 31017 raft_consensus.cc:374] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.718302 31017 raft_consensus.cc:729] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.719039 31017 consensus_queue.cc:260] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.718330 30610 catalog_manager.cc:5495] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "e0691006952a44cfa5f2397f172bfcf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:09.709276 31063 consensus_queue.cc:237] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } }
I20260805 22:23:09.728425 31017 ts_tablet_manager.cc:1428] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3: Time spent starting tablet: real 0.043s user 0.003s sys 0.003s
I20260805 22:23:09.729439 31017 tablet_bootstrap.cc:492] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3: Bootstrap starting.
I20260805 22:23:09.727120 31070 raft_consensus.cc:504] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.736778 30961 raft_consensus.cc:2379] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e0691006952a44cfa5f2397f172bfcf3 in current term 1: Already voted for candidate 5366b554194f4acdb8916d0b1b2289e4 in this term.
I20260805 22:23:09.738322 31014 tablet_bootstrap.cc:492] T 0a6d6d5cabf34c998bba078b482c0b05 P 5366b554194f4acdb8916d0b1b2289e4: No bootstrap required, opened a new log
I20260805 22:23:09.738788 31014 ts_tablet_manager.cc:1397] T 0a6d6d5cabf34c998bba078b482c0b05 P 5366b554194f4acdb8916d0b1b2289e4: Time spent bootstrapping tablet: real 0.072s user 0.013s sys 0.003s
I20260805 22:23:09.738808 30820 leader_election.cc:302] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e0691006952a44cfa5f2397f172bfcf3; no voters: 5366b554194f4acdb8916d0b1b2289e4, 9f55007ac96c440da32b0ef9e5e29281
I20260805 22:23:09.740017 31067 raft_consensus.cc:2726] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260805 22:23:09.733533 30963 raft_consensus.cc:2454] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 1.
W20260805 22:23:09.743336 31077 log.cc:924] Time spent T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260805 22:23:09.743782 31017 tablet_bootstrap.cc:654] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.741922 31014 raft_consensus.cc:348] T 0a6d6d5cabf34c998bba078b482c0b05 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.745122 31014 raft_consensus.cc:374] T 0a6d6d5cabf34c998bba078b482c0b05 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.745313 31016 tablet_bootstrap.cc:492] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281: No bootstrap required, opened a new log
I20260805 22:23:09.745446 31014 raft_consensus.cc:729] T 0a6d6d5cabf34c998bba078b482c0b05 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.745857 31016 ts_tablet_manager.cc:1397] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281: Time spent bootstrapping tablet: real 0.156s user 0.012s sys 0.002s
I20260805 22:23:09.745817 30820 leader_election.cc:302] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3; no voters: 9f55007ac96c440da32b0ef9e5e29281
I20260805 22:23:09.746230 31014 consensus_queue.cc:260] T 0a6d6d5cabf34c998bba078b482c0b05 P 5366b554194f4acdb8916d0b1b2289e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.747148 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:09.748368 31067 raft_consensus.cc:2781] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:09.748562 31014 ts_tablet_manager.cc:1428] T 0a6d6d5cabf34c998bba078b482c0b05 P 5366b554194f4acdb8916d0b1b2289e4: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260805 22:23:09.748616 31016 raft_consensus.cc:348] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
W20260805 22:23:09.749437 30916 leader_election.cc:341] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:09.749557 31014 tablet_bootstrap.cc:492] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4: Bootstrap starting.
I20260805 22:23:09.749544 31016 raft_consensus.cc:374] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.748879 31067 raft_consensus.cc:686] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 1 LEADER]: Becoming Leader. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Running, Role: LEADER
I20260805 22:23:09.750018 31016 raft_consensus.cc:729] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.750808 31016 consensus_queue.cc:260] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.750758 31067 consensus_queue.cc:237] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.753364 31016 ts_tablet_manager.cc:1428] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260805 22:23:09.745239 31080 raft_consensus.cc:482] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:09.755924 31016 tablet_bootstrap.cc:492] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281: Bootstrap starting.
I20260805 22:23:09.758100 31014 tablet_bootstrap.cc:654] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.726506 30611 catalog_manager.cc:5495] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "9f55007ac96c440da32b0ef9e5e29281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:09.755415 31080 raft_consensus.cc:504] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
W20260805 22:23:09.754510 30909 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:37397, user_credentials={real_user=slave}} blocked reactor thread for 51636.5us
I20260805 22:23:09.768658 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:09.769585 30860 raft_consensus.cc:2379] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 5366b554194f4acdb8916d0b1b2289e4 in current term 1: Already voted for candidate e0691006952a44cfa5f2397f172bfcf3 in this term.
I20260805 22:23:09.770452 30858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a7e5cf5aec44cf9cec9a2024024c5f" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:09.771145 30858 raft_consensus.cc:3037] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.774382 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a3f906013c4bf3bc7b79a0777c0213" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281" is_pre_election: true
I20260805 22:23:09.775043 30765 raft_consensus.cc:2454] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 0.
I20260805 22:23:09.776032 31016 tablet_bootstrap.cc:654] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.776333 30916 leader_election.cc:302] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, 9f55007ac96c440da32b0ef9e5e29281; no voters:
I20260805 22:23:09.767086 31070 leader_election.cc:288] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397)
I20260805 22:23:09.777212 31020 raft_consensus.cc:2781] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:09.777650 31020 raft_consensus.cc:482] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:09.777974 31020 raft_consensus.cc:3037] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.781657 30909 leader_election.cc:302] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4; no voters: 9f55007ac96c440da32b0ef9e5e29281, e0691006952a44cfa5f2397f172bfcf3
I20260805 22:23:09.783964 31080 leader_election.cc:288] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397)
I20260805 22:23:09.788800 31070 raft_consensus.cc:2726] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260805 22:23:09.794220 31017 tablet_bootstrap.cc:492] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3: No bootstrap required, opened a new log
I20260805 22:23:09.794689 31017 ts_tablet_manager.cc:1397] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3: Time spent bootstrapping tablet: real 0.065s user 0.019s sys 0.006s
I20260805 22:23:09.797252 31017 raft_consensus.cc:348] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.798419 31017 raft_consensus.cc:374] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.798763 31017 raft_consensus.cc:729] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.799602 31017 consensus_queue.cc:260] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.804114 31017 ts_tablet_manager.cc:1428] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260805 22:23:09.805526 31017 tablet_bootstrap.cc:492] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3: Bootstrap starting.
I20260805 22:23:09.807221 30858 raft_consensus.cc:2454] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 1.
I20260805 22:23:09.807605 31014 tablet_bootstrap.cc:492] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4: No bootstrap required, opened a new log
I20260805 22:23:09.808068 31014 ts_tablet_manager.cc:1397] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4: Time spent bootstrapping tablet: real 0.059s user 0.014s sys 0.010s
I20260805 22:23:09.810228 31020 raft_consensus.cc:504] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.811746 31014 raft_consensus.cc:348] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.812438 31014 raft_consensus.cc:374] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.812647 31017 tablet_bootstrap.cc:654] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.812702 31014 raft_consensus.cc:729] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.813032 31016 tablet_bootstrap.cc:492] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281: No bootstrap required, opened a new log
I20260805 22:23:09.813259 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a3f906013c4bf3bc7b79a0777c0213" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:09.813484 31014 consensus_queue.cc:260] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.813989 31020 leader_election.cc:288] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397)
I20260805 22:23:09.814186 30765 raft_consensus.cc:3037] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.815265 30909 leader_election.cc:302] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3; no voters: 9f55007ac96c440da32b0ef9e5e29281
I20260805 22:23:09.815778 31014 ts_tablet_manager.cc:1428] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4: Time spent starting tablet: real 0.008s user 0.005s sys 0.001s
I20260805 22:23:09.815866 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a3f906013c4bf3bc7b79a0777c0213" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:09.816586 30858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a3f906013c4bf3bc7b79a0777c0213" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3" is_pre_election: true
I20260805 22:23:09.817492 31016 ts_tablet_manager.cc:1397] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281: Time spent bootstrapping tablet: real 0.062s user 0.015s sys 0.001s
I20260805 22:23:09.820173 30858 raft_consensus.cc:2454] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 0.
I20260805 22:23:09.819584 31089 raft_consensus.cc:2781] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:09.821584 31089 raft_consensus.cc:686] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 1 LEADER]: Becoming Leader. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Running, Role: LEADER
I20260805 22:23:09.821419 30860 raft_consensus.cc:3037] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.822690 31089 consensus_queue.cc:237] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.830338 30765 raft_consensus.cc:2454] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 1.
I20260805 22:23:09.831689 30916 leader_election.cc:302] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, 9f55007ac96c440da32b0ef9e5e29281; no voters:
I20260805 22:23:09.832762 31063 raft_consensus.cc:2781] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:09.833276 31063 raft_consensus.cc:686] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 LEADER]: Becoming Leader. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Running, Role: LEADER
I20260805 22:23:09.834301 31063 consensus_queue.cc:237] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.835902 31016 raft_consensus.cc:348] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.831741 30860 raft_consensus.cc:2454] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 1.
I20260805 22:23:09.836697 31016 raft_consensus.cc:374] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.837018 31016 raft_consensus.cc:729] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.840380 31016 consensus_queue.cc:260] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.841514 31017 tablet_bootstrap.cc:492] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3: No bootstrap required, opened a new log
I20260805 22:23:09.841977 31017 ts_tablet_manager.cc:1397] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3: Time spent bootstrapping tablet: real 0.037s user 0.013s sys 0.012s
I20260805 22:23:09.844663 31017 raft_consensus.cc:348] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.845155 31017 raft_consensus.cc:374] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.845424 31017 raft_consensus.cc:729] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.846024 31017 consensus_queue.cc:260] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.847359 31017 ts_tablet_manager.cc:1428] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260805 22:23:09.850683 31016 ts_tablet_manager.cc:1428] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281: Time spent starting tablet: real 0.030s user 0.005s sys 0.002s
I20260805 22:23:09.852238 31016 tablet_bootstrap.cc:492] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281: Bootstrap starting.
I20260805 22:23:09.854894 30610 catalog_manager.cc:5495] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 reported cstate change: term changed from 1 to 2, leader changed from 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193) to 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "5366b554194f4acdb8916d0b1b2289e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:09.856470 30610 catalog_manager.cc:5495] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "5366b554194f4acdb8916d0b1b2289e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:09.857823 30610 catalog_manager.cc:5495] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "5366b554194f4acdb8916d0b1b2289e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:09.858724 31016 tablet_bootstrap.cc:654] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.863284 31016 tablet_bootstrap.cc:492] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281: No bootstrap required, opened a new log
I20260805 22:23:09.863613 31016 ts_tablet_manager.cc:1397] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260805 22:23:09.865494 31016 raft_consensus.cc:348] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.865940 31016 raft_consensus.cc:374] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.866151 31016 raft_consensus.cc:729] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.866668 31016 consensus_queue.cc:260] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.868573 31016 ts_tablet_manager.cc:1428] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260805 22:23:09.869405 31016 tablet_bootstrap.cc:492] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281: Bootstrap starting.
I20260805 22:23:09.872970 30611 catalog_manager.cc:5495] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "e0691006952a44cfa5f2397f172bfcf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:09.874125 31016 tablet_bootstrap.cc:654] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:09.879457 31016 tablet_bootstrap.cc:492] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281: No bootstrap required, opened a new log
I20260805 22:23:09.879878 31016 ts_tablet_manager.cc:1397] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260805 22:23:09.882196 31016 raft_consensus.cc:348] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.882817 31016 raft_consensus.cc:374] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:09.883051 31016 raft_consensus.cc:729] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Initialized, Role: FOLLOWER
I20260805 22:23:09.883738 31028 raft_consensus.cc:482] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:09.883679 31016 consensus_queue.cc:260] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.884231 31028 raft_consensus.cc:504] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.886186 31016 ts_tablet_manager.cc:1428] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260805 22:23:09.886253 31028 leader_election.cc:288] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:09.886813 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa4b450242b446fb1884b71a6bf55e5" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3" is_pre_election: true
I20260805 22:23:09.887264 30860 raft_consensus.cc:2454] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 0.
I20260805 22:23:09.887651 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa4b450242b446fb1884b71a6bf55e5" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4" is_pre_election: true
I20260805 22:23:09.888161 30709 leader_election.cc:302] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:09.888264 30963 raft_consensus.cc:2454] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 0.
I20260805 22:23:09.888958 31028 raft_consensus.cc:2781] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:09.889436 31028 raft_consensus.cc:482] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:09.889766 31028 raft_consensus.cc:3037] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.895551 31028 raft_consensus.cc:504] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.897025 31028 leader_election.cc:288] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 election: Requested vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:09.897914 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa4b450242b446fb1884b71a6bf55e5" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:09.898052 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa4b450242b446fb1884b71a6bf55e5" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:09.898541 30860 raft_consensus.cc:3037] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.898559 30963 raft_consensus.cc:3037] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.905074 30963 raft_consensus.cc:2454] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 1.
I20260805 22:23:09.905074 30860 raft_consensus.cc:2454] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 1.
I20260805 22:23:09.906330 30716 leader_election.cc:302] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, 9f55007ac96c440da32b0ef9e5e29281; no voters:
I20260805 22:23:09.906984 31028 raft_consensus.cc:2781] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:09.907437 31028 raft_consensus.cc:686] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 LEADER]: Becoming Leader. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Running, Role: LEADER
I20260805 22:23:09.908044 31028 consensus_queue.cc:237] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.909773 31065 raft_consensus.cc:482] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:09.910169 31065 raft_consensus.cc:504] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.912488 31065 leader_election.cc:288] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:09.913199 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a7e5cf5aec44cf9cec9a2024024c5f" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3" is_pre_election: true
I20260805 22:23:09.914000 30860 raft_consensus.cc:2379] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f55007ac96c440da32b0ef9e5e29281 in current term 1: Already voted for candidate 5366b554194f4acdb8916d0b1b2289e4 in this term.
I20260805 22:23:09.914575 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a7e5cf5aec44cf9cec9a2024024c5f" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4" is_pre_election: true
I20260805 22:23:09.915786 30716 leader_election.cc:302] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281; no voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3
I20260805 22:23:09.916450 31065 raft_consensus.cc:3037] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.916213 30610 catalog_manager.cc:5495] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "9f55007ac96c440da32b0ef9e5e29281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:09.921844 31065 raft_consensus.cc:2726] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:09.955090 31065 raft_consensus.cc:482] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:09.955547 31065 raft_consensus.cc:504] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.957126 31065 leader_election.cc:288] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:09.958050 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6d6d5cabf34c998bba078b482c0b05" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3" is_pre_election: true
I20260805 22:23:09.958154 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6d6d5cabf34c998bba078b482c0b05" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4" is_pre_election: true
I20260805 22:23:09.958570 30860 raft_consensus.cc:2454] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 0.
I20260805 22:23:09.958702 30963 raft_consensus.cc:2454] T 0a6d6d5cabf34c998bba078b482c0b05 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 0.
I20260805 22:23:09.959427 30709 leader_election.cc:302] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:09.960103 31065 raft_consensus.cc:2781] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:09.960391 31065 raft_consensus.cc:482] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:09.960618 31065 raft_consensus.cc:3037] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.965066 31065 raft_consensus.cc:504] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.966589 31065 leader_election.cc:288] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 election: Requested vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:09.967482 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6d6d5cabf34c998bba078b482c0b05" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:09.967603 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a6d6d5cabf34c998bba078b482c0b05" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:09.967945 30860 raft_consensus.cc:3037] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.968168 30963 raft_consensus.cc:3037] T 0a6d6d5cabf34c998bba078b482c0b05 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:09.973029 30963 raft_consensus.cc:2454] T 0a6d6d5cabf34c998bba078b482c0b05 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 1.
I20260805 22:23:09.973768 30860 raft_consensus.cc:2454] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 1.
I20260805 22:23:09.974012 30716 leader_election.cc:302] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, 9f55007ac96c440da32b0ef9e5e29281; no voters:
I20260805 22:23:09.974751 31065 raft_consensus.cc:2781] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:09.975119 31065 raft_consensus.cc:686] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 LEADER]: Becoming Leader. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Running, Role: LEADER
I20260805 22:23:09.975824 31065 consensus_queue.cc:237] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:09.983454 30610 catalog_manager.cc:5495] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "9f55007ac96c440da32b0ef9e5e29281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:10.049202 31068 raft_consensus.cc:482] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:10.049741 31068 raft_consensus.cc:504] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:10.051470 31068 leader_election.cc:288] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:10.052336 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbd9c17620b4dc1bfc9cd69188253e9" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281" is_pre_election: true
I20260805 22:23:10.052706 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbd9c17620b4dc1bfc9cd69188253e9" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4" is_pre_election: true
I20260805 22:23:10.053010 30765 raft_consensus.cc:2454] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 0.
I20260805 22:23:10.053288 30963 raft_consensus.cc:2454] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 0.
I20260805 22:23:10.054186 30820 leader_election.cc:302] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:10.054112 31068 consensus_queue.cc:1035] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:10.054865 31067 raft_consensus.cc:2781] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:10.055227 31067 raft_consensus.cc:482] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:10.055478 31067 raft_consensus.cc:3037] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:10.061148 31067 raft_consensus.cc:504] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:10.063114 31067 leader_election.cc:288] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:10.064795 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbd9c17620b4dc1bfc9cd69188253e9" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:10.065215 30764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbd9c17620b4dc1bfc9cd69188253e9" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:10.065440 30963 raft_consensus.cc:3037] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:10.065841 30764 raft_consensus.cc:3037] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:10.071965 30963 raft_consensus.cc:2454] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 1.
I20260805 22:23:10.071797 31024 consensus_queue.cc:1035] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:10.073019 30820 leader_election.cc:302] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:10.073133 30764 raft_consensus.cc:2454] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 1.
I20260805 22:23:10.073891 31024 raft_consensus.cc:2781] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:10.075400 30860 raft_consensus.cc:1226] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Rejecting Update request from peer 9f55007ac96c440da32b0ef9e5e29281 for earlier term 1. Current term is 2. Ops: []
I20260805 22:23:10.075688 31024 raft_consensus.cc:686] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 1 LEADER]: Becoming Leader. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Running, Role: LEADER
I20260805 22:23:10.076815 31024 consensus_queue.cc:237] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:10.077797 31028 consensus_queue.cc:1046] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:10.079376 31028 raft_consensus.cc:3032] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:23:10.079622 31028 raft_consensus.cc:729] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Running, Role: LEADER
I20260805 22:23:10.080377 31028 consensus_queue.cc:260] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } }
I20260805 22:23:10.081652 31028 raft_consensus.cc:3037] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:10.088001 31065 raft_consensus.cc:482] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:10.088549 31065 raft_consensus.cc:504] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:10.087875 30610 catalog_manager.cc:5495] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194). New cstate: current_term: 1 leader_uuid: "e0691006952a44cfa5f2397f172bfcf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:10.091523 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3" is_pre_election: true
I20260805 22:23:10.092165 30860 raft_consensus.cc:2379] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f55007ac96c440da32b0ef9e5e29281 in current term 1: Already voted for candidate e0691006952a44cfa5f2397f172bfcf3 in this term.
I20260805 22:23:10.094249 31065 leader_election.cc:288] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:10.095306 30961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4" is_pre_election: true
I20260805 22:23:10.096036 30961 raft_consensus.cc:2379] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f55007ac96c440da32b0ef9e5e29281 in current term 1: Already voted for candidate 5366b554194f4acdb8916d0b1b2289e4 in this term.
I20260805 22:23:10.097076 30716 leader_election.cc:302] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281; no voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3
I20260805 22:23:10.097831 31065 raft_consensus.cc:3037] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:10.104211 31065 raft_consensus.cc:2726] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:10.153431 30765 raft_consensus.cc:1261] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Refusing update from remote peer 5366b554194f4acdb8916d0b1b2289e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260805 22:23:10.153796 30765 pending_rounds.cc:76] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281: Aborting all ops after (but not including) 0
I20260805 22:23:10.153999 30765 pending_rounds.cc:97] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260805 22:23:10.154258 30765 raft_consensus.cc:2866] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:23:10.155539 31089 consensus_queue.cc:1035] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:10.162797 30860 raft_consensus.cc:1261] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Refusing update from remote peer 5366b554194f4acdb8916d0b1b2289e4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260805 22:23:10.164474 31063 consensus_queue.cc:1035] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:10.191836 31063 consensus_queue.cc:1035] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:10.201761 31063 consensus_queue.cc:1035] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:10.258940 31063 consensus_queue.cc:1035] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:10.269173 31063 consensus_queue.cc:1035] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:10.276965 31065 raft_consensus.cc:482] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:10.277479 31065 raft_consensus.cc:504] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:10.278949 31065 leader_election.cc:288] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:10.279784 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de85cd15b9354ebbbd8be638aa855495" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0691006952a44cfa5f2397f172bfcf3" is_pre_election: true
I20260805 22:23:10.280036 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de85cd15b9354ebbbd8be638aa855495" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4" is_pre_election: true
I20260805 22:23:10.280726 30963 raft_consensus.cc:2379] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9f55007ac96c440da32b0ef9e5e29281 in current term 1: Already voted for candidate e0691006952a44cfa5f2397f172bfcf3 in this term.
I20260805 22:23:10.281616 30716 leader_election.cc:302] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281; no voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3
I20260805 22:23:10.282428 31065 raft_consensus.cc:3037] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:10.288398 31065 raft_consensus.cc:2726] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:10.321048 31065 consensus_queue.cc:1035] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:10.331560 31111 consensus_queue.cc:1035] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:10.371062 31024 consensus_queue.cc:1035] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:10.374155 31024 consensus_queue.cc:1035] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:10.400605 31111 consensus_queue.cc:1035] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:10.411269 31111 consensus_queue.cc:1035] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:10.417560 31063 consensus_queue.cc:1035] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:10.429116 31063 consensus_queue.cc:1035] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260805 22:23:10.508028 30685 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260805 22:23:10.594624 31044 consensus_queue.cc:1035] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:10.605824 31068 consensus_queue.cc:1035] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:11.245167 31044 raft_consensus.cc:482] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:11.245697 31044 raft_consensus.cc:504] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:11.247370 31044 leader_election.cc:288] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:11.248212 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281" is_pre_election: true
I20260805 22:23:11.248458 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4" is_pre_election: true
I20260805 22:23:11.248975 30765 raft_consensus.cc:2454] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 1.
I20260805 22:23:11.249044 30963 raft_consensus.cc:2454] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 1.
I20260805 22:23:11.250161 30820 leader_election.cc:302] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:11.250810 31044 raft_consensus.cc:2781] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:23:11.251127 31044 raft_consensus.cc:482] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:11.251365 31044 raft_consensus.cc:3037] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:11.256083 31044 raft_consensus.cc:504] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:11.257776 31044 leader_election.cc:288] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 2 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:11.258862 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:11.259047 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:11.259615 30765 raft_consensus.cc:3037] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:11.259642 30963 raft_consensus.cc:3037] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:11.266522 30963 raft_consensus.cc:2454] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 2.
I20260805 22:23:11.266515 30765 raft_consensus.cc:2454] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 2.
I20260805 22:23:11.267731 30820 leader_election.cc:302] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:11.268446 31044 raft_consensus.cc:2781] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:23:11.268868 31044 raft_consensus.cc:686] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 2 LEADER]: Becoming Leader. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Running, Role: LEADER
I20260805 22:23:11.269668 31044 consensus_queue.cc:237] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:11.278102 30610 catalog_manager.cc:5495] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 reported cstate change: term changed from 0 to 2, leader changed from <none> to e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "e0691006952a44cfa5f2397f172bfcf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:11.670595 30765 raft_consensus.cc:1261] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Refusing update from remote peer e0691006952a44cfa5f2397f172bfcf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260805 22:23:11.671819 31044 consensus_queue.cc:1035] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:11.683544 30961 raft_consensus.cc:1261] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Refusing update from remote peer e0691006952a44cfa5f2397f172bfcf3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260805 22:23:11.684900 31024 consensus_queue.cc:1035] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:11.770779 30961 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bd2a35893d484eae80fe9e16ba237fcc"
dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
mode: GRACEFUL
new_leader_uuid: "9f55007ac96c440da32b0ef9e5e29281"
from {username='slave'} at 127.0.0.1:55470
I20260805 22:23:11.771413 30961 raft_consensus.cc:595] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 2 LEADER]: Received request to transfer leadership to TS 9f55007ac96c440da32b0ef9e5e29281
I20260805 22:23:11.794062 30858 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c"
dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
mode: GRACEFUL
new_leader_uuid: "9f55007ac96c440da32b0ef9e5e29281"
from {username='slave'} at 127.0.0.1:45208
I20260805 22:23:11.794803 30858 raft_consensus.cc:595] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 2 LEADER]: Received request to transfer leadership to TS 9f55007ac96c440da32b0ef9e5e29281
I20260805 22:23:11.800097 30858 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5bbd9c17620b4dc1bfc9cd69188253e9"
dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
mode: GRACEFUL
new_leader_uuid: "9f55007ac96c440da32b0ef9e5e29281"
from {username='slave'} at 127.0.0.1:45208
I20260805 22:23:11.800604 30858 raft_consensus.cc:595] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 1 LEADER]: Received request to transfer leadership to TS 9f55007ac96c440da32b0ef9e5e29281
I20260805 22:23:11.806092 30961 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "77a3f906013c4bf3bc7b79a0777c0213"
dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
mode: GRACEFUL
new_leader_uuid: "e0691006952a44cfa5f2397f172bfcf3"
from {username='slave'} at 127.0.0.1:55470
I20260805 22:23:11.806700 30961 raft_consensus.cc:595] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 LEADER]: Received request to transfer leadership to TS e0691006952a44cfa5f2397f172bfcf3
I20260805 22:23:11.811602 30961 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5"
dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
mode: GRACEFUL
new_leader_uuid: "e0691006952a44cfa5f2397f172bfcf3"
from {username='slave'} at 127.0.0.1:55470
I20260805 22:23:11.812183 30961 raft_consensus.cc:595] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 LEADER]: Received request to transfer leadership to TS e0691006952a44cfa5f2397f172bfcf3
I20260805 22:23:11.820134 30961 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e9a7e5cf5aec44cf9cec9a2024024c5f"
dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
mode: GRACEFUL
new_leader_uuid: "e0691006952a44cfa5f2397f172bfcf3"
from {username='slave'} at 127.0.0.1:55470
I20260805 22:23:11.820690 30961 raft_consensus.cc:595] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 1 LEADER]: Received request to transfer leadership to TS e0691006952a44cfa5f2397f172bfcf3
I20260805 22:23:11.825568 30858 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8a62b2e2dc4145aca8e0a45586dfe57d"
dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
mode: GRACEFUL
new_leader_uuid: "5366b554194f4acdb8916d0b1b2289e4"
from {username='slave'} at 127.0.0.1:45208
I20260805 22:23:11.826118 30858 raft_consensus.cc:595] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 1 LEADER]: Received request to transfer leadership to TS 5366b554194f4acdb8916d0b1b2289e4
I20260805 22:23:11.874190 31152 raft_consensus.cc:979] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4: : Instructing follower e0691006952a44cfa5f2397f172bfcf3 to start an election
I20260805 22:23:11.874646 31063 raft_consensus.cc:1067] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 LEADER]: Signalling peer e0691006952a44cfa5f2397f172bfcf3 to start an election
I20260805 22:23:11.876222 30858 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77a3f906013c4bf3bc7b79a0777c0213"
dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
from {username='slave'} at 127.0.0.1:45196
I20260805 22:23:11.876780 30858 raft_consensus.cc:482] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:23:11.877126 30858 raft_consensus.cc:3037] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:11.882299 30858 raft_consensus.cc:504] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:11.884016 30858 leader_election.cc:288] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 2 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:11.884972 30764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a3f906013c4bf3bc7b79a0777c0213" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:11.885187 30961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a3f906013c4bf3bc7b79a0777c0213" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:11.885548 30764 raft_consensus.cc:3037] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:11.885751 30961 raft_consensus.cc:3032] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:23:11.886046 30961 raft_consensus.cc:729] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Running, Role: LEADER
I20260805 22:23:11.886718 30961 consensus_queue.cc:260] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:11.887710 30961 raft_consensus.cc:3037] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:11.890533 30764 raft_consensus.cc:2454] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 2.
I20260805 22:23:11.891639 30820 leader_election.cc:302] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:11.892491 31068 raft_consensus.cc:2781] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:23:11.892920 31068 raft_consensus.cc:686] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 2 LEADER]: Becoming Leader. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Running, Role: LEADER
I20260805 22:23:11.892963 30961 raft_consensus.cc:2454] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 2.
I20260805 22:23:11.893857 31068 consensus_queue.cc:237] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:11.901988 30609 catalog_manager.cc:5495] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 reported cstate change: term changed from 1 to 2, leader changed from 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195) to e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "e0691006952a44cfa5f2397f172bfcf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:12.099860 31152 raft_consensus.cc:979] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4: : Instructing follower e0691006952a44cfa5f2397f172bfcf3 to start an election
I20260805 22:23:12.100303 31063 raft_consensus.cc:1067] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 LEADER]: Signalling peer e0691006952a44cfa5f2397f172bfcf3 to start an election
I20260805 22:23:12.101857 30858 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5"
dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
from {username='slave'} at 127.0.0.1:45196
I20260805 22:23:12.102388 30858 raft_consensus.cc:482] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:23:12.102684 30858 raft_consensus.cc:3037] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:12.107479 30858 raft_consensus.cc:504] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:12.108979 30858 leader_election.cc:288] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 2 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:12.109969 30764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:12.110172 30961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:12.110491 30764 raft_consensus.cc:3037] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:12.110716 30961 raft_consensus.cc:3032] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:23:12.111104 30961 raft_consensus.cc:729] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Running, Role: LEADER
I20260805 22:23:12.111726 30961 consensus_queue.cc:260] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:12.112703 30961 raft_consensus.cc:3037] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:12.115320 30764 raft_consensus.cc:2454] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 2.
I20260805 22:23:12.116350 30820 leader_election.cc:302] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281, e0691006952a44cfa5f2397f172bfcf3; no voters:
W20260805 22:23:12.116914 30889 debug-util.cc:398] Leaking SignalData structure 0x7b08000ffd00 after lost signal to thread 29786
I20260805 22:23:12.117769 31068 raft_consensus.cc:2781] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Leader election won for term 2
W20260805 22:23:12.117887 30889 debug-util.cc:398] Leaking SignalData structure 0x7b08000fb420 after lost signal to thread 30668
I20260805 22:23:12.117870 30961 raft_consensus.cc:2454] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 2.
I20260805 22:23:12.118286 31068 raft_consensus.cc:686] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 2 LEADER]: Becoming Leader. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Running, Role: LEADER
W20260805 22:23:12.119130 30889 debug-util.cc:398] Leaking SignalData structure 0x7b0800165240 after lost signal to thread 30797
I20260805 22:23:12.119166 31068 consensus_queue.cc:237] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:12.127151 30609 catalog_manager.cc:5495] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 reported cstate change: term changed from 1 to 2, leader changed from 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195) to e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "e0691006952a44cfa5f2397f172bfcf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:12.187665 31044 raft_consensus.cc:979] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3: : Instructing follower 5366b554194f4acdb8916d0b1b2289e4 to start an election
I20260805 22:23:12.188122 31068 raft_consensus.cc:1067] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 1 LEADER]: Signalling peer 5366b554194f4acdb8916d0b1b2289e4 to start an election
I20260805 22:23:12.189978 30961 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8a62b2e2dc4145aca8e0a45586dfe57d"
dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
from {username='slave'} at 127.0.0.1:55446
I20260805 22:23:12.190587 30961 raft_consensus.cc:482] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:23:12.190866 30961 raft_consensus.cc:3037] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:12.196059 30961 raft_consensus.cc:504] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:12.197821 30961 leader_election.cc:288] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 2 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397)
I20260805 22:23:12.198966 30858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a62b2e2dc4145aca8e0a45586dfe57d" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:12.199764 30858 raft_consensus.cc:3032] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:23:12.200102 30858 raft_consensus.cc:729] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Running, Role: LEADER
I20260805 22:23:12.200909 30858 consensus_queue.cc:260] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:12.202356 30858 raft_consensus.cc:3037] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:12.198968 30764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a62b2e2dc4145aca8e0a45586dfe57d" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:12.205711 30764 raft_consensus.cc:3037] T 8a62b2e2dc4145aca8e0a45586dfe57d P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:12.208568 30858 raft_consensus.cc:2454] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 2.
I20260805 22:23:12.209945 31152 raft_consensus.cc:979] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4: : Instructing follower 9f55007ac96c440da32b0ef9e5e29281 to start an election
I20260805 22:23:12.211292 30909 leader_election.cc:302] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:12.213054 30764 raft_consensus.cc:2454] T 8a62b2e2dc4145aca8e0a45586dfe57d P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 2.
I20260805 22:23:12.214708 31163 raft_consensus.cc:1067] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 2 LEADER]: Signalling peer 9f55007ac96c440da32b0ef9e5e29281 to start an election
I20260805 22:23:12.215191 31063 raft_consensus.cc:2781] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:23:12.215632 31063 raft_consensus.cc:686] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [term 2 LEADER]: Becoming Leader. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Running, Role: LEADER
I20260805 22:23:12.216471 31063 consensus_queue.cc:237] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:12.219102 30764 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bd2a35893d484eae80fe9e16ba237fcc"
dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
from {username='slave'} at 127.0.0.1:54034
I20260805 22:23:12.219758 30764 raft_consensus.cc:482] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:23:12.219965 31044 raft_consensus.cc:979] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3: : Instructing follower 9f55007ac96c440da32b0ef9e5e29281 to start an election
I20260805 22:23:12.220157 30764 raft_consensus.cc:3037] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:23:12.220440 31044 raft_consensus.cc:1067] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 1 LEADER]: Signalling peer 9f55007ac96c440da32b0ef9e5e29281 to start an election
I20260805 22:23:12.222546 30765 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5bbd9c17620b4dc1bfc9cd69188253e9"
dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
from {username='slave'} at 127.0.0.1:54030
I20260805 22:23:12.223156 30765 raft_consensus.cc:482] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:23:12.223582 30765 raft_consensus.cc:3037] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:12.229028 30764 raft_consensus.cc:504] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } }
I20260805 22:23:12.230455 30765 raft_consensus.cc:504] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:12.232180 30764 leader_election.cc:288] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 3 election: Requested vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:12.232302 30858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2a35893d484eae80fe9e16ba237fcc" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:12.232947 30858 raft_consensus.cc:3037] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:23:12.233280 30961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd2a35893d484eae80fe9e16ba237fcc" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:12.233877 30961 raft_consensus.cc:3032] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:23:12.230633 30609 catalog_manager.cc:5495] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 reported cstate change: term changed from 1 to 2, leader changed from e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194) to 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "5366b554194f4acdb8916d0b1b2289e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:12.235702 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbd9c17620b4dc1bfc9cd69188253e9" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:12.236269 30860 raft_consensus.cc:3032] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:23:12.236605 30860 raft_consensus.cc:729] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Running, Role: LEADER
I20260805 22:23:12.237370 30860 consensus_queue.cc:260] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:12.238517 30860 raft_consensus.cc:3037] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:12.234184 30961 raft_consensus.cc:729] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Running, Role: LEADER
I20260805 22:23:12.240149 30858 raft_consensus.cc:2454] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 3.
I20260805 22:23:12.240926 30765 leader_election.cc:288] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 2 election: Requested vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:12.240211 30961 consensus_queue.cc:260] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } }
I20260805 22:23:12.242775 30961 raft_consensus.cc:3037] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:23:12.243678 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbd9c17620b4dc1bfc9cd69188253e9" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:12.244258 30963 raft_consensus.cc:3037] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:12.242723 30709 leader_election.cc:302] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:12.248981 31111 raft_consensus.cc:2781] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:23:12.249811 30860 raft_consensus.cc:2454] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 2.
I20260805 22:23:12.250013 30961 raft_consensus.cc:2454] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 3.
I20260805 22:23:12.253067 30963 raft_consensus.cc:2454] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 2.
I20260805 22:23:12.254984 30716 leader_election.cc:302] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, 9f55007ac96c440da32b0ef9e5e29281; no voters:
I20260805 22:23:12.258497 31063 raft_consensus.cc:979] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4: : Instructing follower e0691006952a44cfa5f2397f172bfcf3 to start an election
I20260805 22:23:12.258975 31063 raft_consensus.cc:1067] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 1 LEADER]: Signalling peer e0691006952a44cfa5f2397f172bfcf3 to start an election
I20260805 22:23:12.260977 30858 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e9a7e5cf5aec44cf9cec9a2024024c5f"
dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
from {username='slave'} at 127.0.0.1:45196
I20260805 22:23:12.261673 30858 raft_consensus.cc:482] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:23:12.262071 30858 raft_consensus.cc:3037] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:12.264950 31165 raft_consensus.cc:2781] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:23:12.265534 31165 raft_consensus.cc:686] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 LEADER]: Becoming Leader. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Running, Role: LEADER
I20260805 22:23:12.266476 31165 consensus_queue.cc:237] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:12.268603 31111 raft_consensus.cc:686] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Becoming Leader. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Running, Role: LEADER
I20260805 22:23:12.269670 31111 consensus_queue.cc:237] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } }
I20260805 22:23:12.270350 31068 raft_consensus.cc:979] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3: : Instructing follower 9f55007ac96c440da32b0ef9e5e29281 to start an election
I20260805 22:23:12.270920 31068 raft_consensus.cc:1067] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 2 LEADER]: Signalling peer 9f55007ac96c440da32b0ef9e5e29281 to start an election
I20260805 22:23:12.270416 30858 raft_consensus.cc:504] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:12.272881 30858 leader_election.cc:288] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 2 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:12.273999 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a7e5cf5aec44cf9cec9a2024024c5f" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:12.274633 30963 raft_consensus.cc:3032] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:23:12.274964 30963 raft_consensus.cc:729] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Running, Role: LEADER
I20260805 22:23:12.275910 30764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9a7e5cf5aec44cf9cec9a2024024c5f" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:12.276618 30764 raft_consensus.cc:3037] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:12.275882 30963 consensus_queue.cc:260] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:12.278813 30961 raft_consensus.cc:1261] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Refusing update from remote peer e0691006952a44cfa5f2397f172bfcf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:23:12.276167 30765 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c"
dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
from {username='slave'} at 127.0.0.1:54030
I20260805 22:23:12.280339 30765 raft_consensus.cc:482] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:23:12.280440 31044 consensus_queue.cc:1035] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:12.279842 30963 raft_consensus.cc:3037] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:12.280817 30765 raft_consensus.cc:3037] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:23:12.284782 30764 raft_consensus.cc:2454] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 2.
I20260805 22:23:12.285974 30820 leader_election.cc:302] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:12.287284 31044 raft_consensus.cc:2781] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:23:12.287757 31044 raft_consensus.cc:686] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [term 2 LEADER]: Becoming Leader. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Running, Role: LEADER
I20260805 22:23:12.288568 31044 consensus_queue.cc:237] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:12.297567 30963 raft_consensus.cc:2454] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 2.
I20260805 22:23:12.300809 30764 raft_consensus.cc:1261] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Refusing update from remote peer e0691006952a44cfa5f2397f172bfcf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:23:12.300614 30609 catalog_manager.cc:5495] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 reported cstate change: term changed from 2 to 3, leader changed from 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195) to 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "9f55007ac96c440da32b0ef9e5e29281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:12.303359 31024 consensus_queue.cc:1035] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:12.317203 30765 raft_consensus.cc:504] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:12.321758 30858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:12.322525 30858 raft_consensus.cc:3032] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:23:12.322866 30858 raft_consensus.cc:729] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Running, Role: LEADER
I20260805 22:23:12.323688 30858 consensus_queue.cc:260] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:12.324913 30858 raft_consensus.cc:3037] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:23:12.324342 30609 catalog_manager.cc:5495] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 reported cstate change: term changed from 1 to 2, leader changed from 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195) to e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194). New cstate: current_term: 2 leader_uuid: "e0691006952a44cfa5f2397f172bfcf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:12.332157 30858 raft_consensus.cc:2454] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 3.
I20260805 22:23:12.340397 30765 leader_election.cc:288] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 3 election: Requested vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:12.343061 30709 leader_election.cc:302] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:12.343984 31165 raft_consensus.cc:2781] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:23:12.344470 31165 raft_consensus.cc:686] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Becoming Leader. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Running, Role: LEADER
I20260805 22:23:12.345414 31165 consensus_queue.cc:237] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:12.346997 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: true dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:12.347622 30963 raft_consensus.cc:3037] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:23:12.344719 30610 catalog_manager.cc:5495] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 reported cstate change: term changed from 1 to 2, leader changed from e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194) to 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193). New cstate: current_term: 2 leader_uuid: "9f55007ac96c440da32b0ef9e5e29281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:12.357370 30963 raft_consensus.cc:2454] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 3.
I20260805 22:23:12.376396 30610 catalog_manager.cc:5495] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 reported cstate change: term changed from 2 to 3, leader changed from e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194) to 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "9f55007ac96c440da32b0ef9e5e29281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:12.595992 30764 raft_consensus.cc:1261] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Refusing update from remote peer e0691006952a44cfa5f2397f172bfcf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:23:12.597335 31044 consensus_queue.cc:1035] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:12.607496 30961 raft_consensus.cc:1261] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Refusing update from remote peer e0691006952a44cfa5f2397f172bfcf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:23:12.609911 31068 consensus_queue.cc:1035] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:12.661572 30961 raft_consensus.cc:1261] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Refusing update from remote peer 9f55007ac96c440da32b0ef9e5e29281: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:23:12.662986 31165 consensus_queue.cc:1035] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:12.663806 30858 raft_consensus.cc:1261] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Refusing update from remote peer 9f55007ac96c440da32b0ef9e5e29281: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:23:12.665112 31165 consensus_queue.cc:1035] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:12.693934 30858 raft_consensus.cc:1261] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Refusing update from remote peer 9f55007ac96c440da32b0ef9e5e29281: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260805 22:23:12.695406 31165 consensus_queue.cc:1035] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:12.704416 30961 raft_consensus.cc:1261] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Refusing update from remote peer 9f55007ac96c440da32b0ef9e5e29281: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260805 22:23:12.706048 31166 consensus_queue.cc:1035] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:12.747833 30961 raft_consensus.cc:1261] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Refusing update from remote peer e0691006952a44cfa5f2397f172bfcf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:23:12.749064 31044 consensus_queue.cc:1035] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:12.752715 30860 raft_consensus.cc:1261] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Refusing update from remote peer 5366b554194f4acdb8916d0b1b2289e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:23:12.754349 31162 consensus_queue.cc:1035] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:12.758188 30764 raft_consensus.cc:1261] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Refusing update from remote peer e0691006952a44cfa5f2397f172bfcf3: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:23:12.763150 31024 consensus_queue.cc:1035] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:12.779435 30765 raft_consensus.cc:1261] T 8a62b2e2dc4145aca8e0a45586dfe57d P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Refusing update from remote peer 5366b554194f4acdb8916d0b1b2289e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:23:12.781777 31063 consensus_queue.cc:1035] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:12.814450 30860 raft_consensus.cc:1261] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Refusing update from remote peer 9f55007ac96c440da32b0ef9e5e29281: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260805 22:23:12.815583 31166 consensus_queue.cc:1035] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:12.825245 30961 raft_consensus.cc:1261] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Refusing update from remote peer 9f55007ac96c440da32b0ef9e5e29281: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260805 22:23:12.826736 31165 consensus_queue.cc:1035] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:14.858460 29783 auto_leader_rebalancer-test.cc:339] The leader distribution is
I20260805 22:23:14.859164 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:14.888378 30765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c"
dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
mode: GRACEFUL
new_leader_uuid: "5366b554194f4acdb8916d0b1b2289e4"
from {username='slave'} at 127.0.0.1:50950
I20260805 22:23:14.888919 30765 raft_consensus.cc:595] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Received request to transfer leadership to TS 5366b554194f4acdb8916d0b1b2289e4
I20260805 22:23:14.892130 30765 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eaa4b450242b446fb1884b71a6bf55e5"
dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
mode: GRACEFUL
new_leader_uuid: "5366b554194f4acdb8916d0b1b2289e4"
from {username='slave'} at 127.0.0.1:50950
I20260805 22:23:14.892894 30765 raft_consensus.cc:595] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 LEADER]: Received request to transfer leadership to TS 5366b554194f4acdb8916d0b1b2289e4
I20260805 22:23:14.894208 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260805 22:23:14.894665 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:14.965969 31111 raft_consensus.cc:979] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281: : Instructing follower 5366b554194f4acdb8916d0b1b2289e4 to start an election
I20260805 22:23:14.967163 31200 raft_consensus.cc:1067] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 LEADER]: Signalling peer 5366b554194f4acdb8916d0b1b2289e4 to start an election
I20260805 22:23:14.969122 30963 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eaa4b450242b446fb1884b71a6bf55e5"
dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
from {username='slave'} at 127.0.0.1:55454
I20260805 22:23:14.969606 30963 raft_consensus.cc:482] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:23:14.969868 30963 raft_consensus.cc:3037] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:14.974269 30963 raft_consensus.cc:504] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:14.975703 30963 leader_election.cc:288] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 2 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397)
I20260805 22:23:14.976999 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa4b450242b446fb1884b71a6bf55e5" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:14.976928 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa4b450242b446fb1884b71a6bf55e5" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:14.977692 30765 raft_consensus.cc:3032] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:23:14.977701 30860 raft_consensus.cc:3037] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:14.978142 30765 raft_consensus.cc:729] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Running, Role: LEADER
I20260805 22:23:14.978916 30765 consensus_queue.cc:260] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:14.980145 30765 raft_consensus.cc:3037] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:14.984390 30860 raft_consensus.cc:2454] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 2.
I20260805 22:23:14.985543 30909 leader_election.cc:302] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:14.986215 30765 raft_consensus.cc:2454] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 2.
I20260805 22:23:14.986310 31196 raft_consensus.cc:2781] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:23:14.988188 31196 raft_consensus.cc:686] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 LEADER]: Becoming Leader. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Running, Role: LEADER
I20260805 22:23:14.989046 31196 consensus_queue.cc:237] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:14.997263 30611 catalog_manager.cc:5495] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 reported cstate change: term changed from 1 to 2, leader changed from 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193) to 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "5366b554194f4acdb8916d0b1b2289e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:15.323251 31201 raft_consensus.cc:979] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281: : Instructing follower 5366b554194f4acdb8916d0b1b2289e4 to start an election
I20260805 22:23:15.323664 31200 raft_consensus.cc:1067] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Signalling peer 5366b554194f4acdb8916d0b1b2289e4 to start an election
I20260805 22:23:15.325002 30963 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c"
dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
from {username='slave'} at 127.0.0.1:55454
I20260805 22:23:15.325464 30963 raft_consensus.cc:482] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:23:15.325760 30963 raft_consensus.cc:3037] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:23:15.330374 30963 raft_consensus.cc:504] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:15.331915 30963 leader_election.cc:288] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 4 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397)
I20260805 22:23:15.332746 30765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:15.333007 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: true dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:15.333348 30765 raft_consensus.cc:3032] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:23:15.333638 30860 raft_consensus.cc:3037] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:23:15.333698 30765 raft_consensus.cc:729] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Running, Role: LEADER
I20260805 22:23:15.334492 30765 consensus_queue.cc:260] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:15.335505 30765 raft_consensus.cc:3037] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:23:15.339233 30860 raft_consensus.cc:2454] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 4.
I20260805 22:23:15.339949 30765 raft_consensus.cc:2454] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 4.
I20260805 22:23:15.340108 30909 leader_election.cc:302] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:15.340819 31196 raft_consensus.cc:2781] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:23:15.341295 31196 raft_consensus.cc:686] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 4 LEADER]: Becoming Leader. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Running, Role: LEADER
I20260805 22:23:15.342056 31196 consensus_queue.cc:237] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:15.348013 30611 catalog_manager.cc:5495] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 reported cstate change: term changed from 3 to 4, leader changed from 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193) to 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195). New cstate: current_term: 4 leader_uuid: "5366b554194f4acdb8916d0b1b2289e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:15.387394 30765 raft_consensus.cc:1261] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Refusing update from remote peer 5366b554194f4acdb8916d0b1b2289e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:23:15.389160 31196 consensus_queue.cc:1035] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:15.399255 30860 raft_consensus.cc:1261] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Refusing update from remote peer 5366b554194f4acdb8916d0b1b2289e4: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:23:15.400542 31202 consensus_queue.cc:1035] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:15.752992 30860 raft_consensus.cc:1261] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 4 FOLLOWER]: Refusing update from remote peer 5366b554194f4acdb8916d0b1b2289e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:23:15.754343 31202 consensus_queue.cc:1035] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:15.763847 30765 raft_consensus.cc:1261] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 4 FOLLOWER]: Refusing update from remote peer 5366b554194f4acdb8916d0b1b2289e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:23:15.765436 31213 consensus_queue.cc:1035] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:15.896011 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:15.902632 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:15.903064 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:16.903880 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:16.909086 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:16.909503 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:17.910298 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:17.916383 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:17.916923 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:18.917909 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:18.923261 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:18.923722 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:19.924479 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:19.929602 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:19.930011 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:20.930907 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:20.936108 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:20.936558 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:21.937476 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:21.942603 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:21.943073 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:22.943859 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:22.948925 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:22.949409 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:23.950336 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:23.955531 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:23.955931 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:24.986181 29783 auto_leader_rebalancer-test.cc:358] The leader distribution is
I20260805 22:23:24.992722 30963 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eaa4b450242b446fb1884b71a6bf55e5"
dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
mode: GRACEFUL
new_leader_uuid: "9f55007ac96c440da32b0ef9e5e29281"
from {username='slave'} at 127.0.0.1:55470
I20260805 22:23:24.993372 30963 raft_consensus.cc:595] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 LEADER]: Received request to transfer leadership to TS 9f55007ac96c440da32b0ef9e5e29281
I20260805 22:23:25.000514 30963 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c"
dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
mode: GRACEFUL
new_leader_uuid: "9f55007ac96c440da32b0ef9e5e29281"
from {username='slave'} at 127.0.0.1:55470
I20260805 22:23:25.000981 30963 raft_consensus.cc:595] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 4 LEADER]: Received request to transfer leadership to TS 9f55007ac96c440da32b0ef9e5e29281
I20260805 22:23:25.008304 30860 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "77a3f906013c4bf3bc7b79a0777c0213"
dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
mode: GRACEFUL
new_leader_uuid: "9f55007ac96c440da32b0ef9e5e29281"
from {username='slave'} at 127.0.0.1:45208
I20260805 22:23:25.008751 30860 raft_consensus.cc:595] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 2 LEADER]: Received request to transfer leadership to TS 9f55007ac96c440da32b0ef9e5e29281
I20260805 22:23:25.013190 30860 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5"
dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
mode: GRACEFUL
new_leader_uuid: "9f55007ac96c440da32b0ef9e5e29281"
from {username='slave'} at 127.0.0.1:45208
I20260805 22:23:25.013716 30860 raft_consensus.cc:595] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 2 LEADER]: Received request to transfer leadership to TS 9f55007ac96c440da32b0ef9e5e29281
I20260805 22:23:25.018451 30860 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "de85cd15b9354ebbbd8be638aa855495"
dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
mode: GRACEFUL
new_leader_uuid: "9f55007ac96c440da32b0ef9e5e29281"
from {username='slave'} at 127.0.0.1:45208
I20260805 22:23:25.018961 30860 raft_consensus.cc:595] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 1 LEADER]: Received request to transfer leadership to TS 9f55007ac96c440da32b0ef9e5e29281
I20260805 22:23:25.019418 31231 raft_consensus.cc:979] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3: : Instructing follower 9f55007ac96c440da32b0ef9e5e29281 to start an election
I20260805 22:23:25.020012 31232 raft_consensus.cc:1067] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 2 LEADER]: Signalling peer 9f55007ac96c440da32b0ef9e5e29281 to start an election
I20260805 22:23:25.021658 30765 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77a3f906013c4bf3bc7b79a0777c0213"
dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
from {username='slave'} at 127.0.0.1:54030
I20260805 22:23:25.022164 30765 raft_consensus.cc:482] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:23:25.022557 30765 raft_consensus.cc:3037] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:23:25.028184 30765 raft_consensus.cc:504] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:25.030294 30765 leader_election.cc:288] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 3 election: Requested vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:25.031262 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a3f906013c4bf3bc7b79a0777c0213" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:25.031075 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a3f906013c4bf3bc7b79a0777c0213" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:25.031756 30963 raft_consensus.cc:3037] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:23:25.031915 30860 raft_consensus.cc:3032] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:23:25.032208 30860 raft_consensus.cc:729] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Running, Role: LEADER
I20260805 22:23:25.032802 30860 consensus_queue.cc:260] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:25.033780 30860 raft_consensus.cc:3037] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:23:25.036248 30963 raft_consensus.cc:2454] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 3.
I20260805 22:23:25.037405 30716 leader_election.cc:302] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, 9f55007ac96c440da32b0ef9e5e29281; no voters:
I20260805 22:23:25.038288 31200 raft_consensus.cc:2781] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:23:25.038739 30860 raft_consensus.cc:2454] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 3.
I20260805 22:23:25.039381 31200 raft_consensus.cc:686] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Becoming Leader. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Running, Role: LEADER
I20260805 22:23:25.040343 31200 consensus_queue.cc:237] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:25.047340 30610 catalog_manager.cc:5495] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 reported cstate change: term changed from 2 to 3, leader changed from e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194) to 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "9f55007ac96c440da32b0ef9e5e29281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:25.102403 31232 raft_consensus.cc:979] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3: : Instructing follower 9f55007ac96c440da32b0ef9e5e29281 to start an election
I20260805 22:23:25.103196 31232 raft_consensus.cc:1067] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 1 LEADER]: Signalling peer 9f55007ac96c440da32b0ef9e5e29281 to start an election
I20260805 22:23:25.104776 30765 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "de85cd15b9354ebbbd8be638aa855495"
dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
from {username='slave'} at 127.0.0.1:54030
I20260805 22:23:25.105296 30765 raft_consensus.cc:482] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:23:25.105706 30765 raft_consensus.cc:3037] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:25.108548 31225 raft_consensus.cc:979] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4: : Instructing follower 9f55007ac96c440da32b0ef9e5e29281 to start an election
I20260805 22:23:25.109133 31241 raft_consensus.cc:1067] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 4 LEADER]: Signalling peer 9f55007ac96c440da32b0ef9e5e29281 to start an election
I20260805 22:23:25.111371 30764 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c"
dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
from {username='slave'} at 127.0.0.1:54034
I20260805 22:23:25.111843 30764 raft_consensus.cc:482] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 4 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:23:25.111740 30765 raft_consensus.cc:504] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:25.112174 30764 raft_consensus.cc:3037] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:23:25.113229 30765 leader_election.cc:288] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 2 election: Requested vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:25.113950 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de85cd15b9354ebbbd8be638aa855495" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:25.114320 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de85cd15b9354ebbbd8be638aa855495" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:25.114498 30860 raft_consensus.cc:3032] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:23:25.114777 30860 raft_consensus.cc:729] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Running, Role: LEADER
I20260805 22:23:25.114807 30963 raft_consensus.cc:3037] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:25.115590 30860 consensus_queue.cc:260] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:25.116503 30860 raft_consensus.cc:3037] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:25.118691 30764 raft_consensus.cc:504] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 5 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:25.120934 30860 raft_consensus.cc:2454] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 2.
I20260805 22:23:25.121449 30963 raft_consensus.cc:2454] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 2.
I20260805 22:23:25.121956 30858 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:25.122099 30961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7960edb18a3a4203b8c683499c8fe92c" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: true dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:25.122481 30716 leader_election.cc:302] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, 9f55007ac96c440da32b0ef9e5e29281; no voters:
I20260805 22:23:25.122695 30961 raft_consensus.cc:3032] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:23:25.122540 30858 raft_consensus.cc:3037] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:23:25.123309 31200 raft_consensus.cc:2781] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:23:25.123709 31200 raft_consensus.cc:686] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 LEADER]: Becoming Leader. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Running, Role: LEADER
I20260805 22:23:25.124403 31200 consensus_queue.cc:237] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:25.123031 30961 raft_consensus.cc:729] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Running, Role: LEADER
I20260805 22:23:25.126183 30961 consensus_queue.cc:260] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:25.127439 30961 raft_consensus.cc:3037] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:23:25.129743 30764 leader_election.cc:288] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 5 election: Requested vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:25.130065 30858 raft_consensus.cc:2454] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 5.
I20260805 22:23:25.131502 30709 leader_election.cc:302] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:25.132390 31200 raft_consensus.cc:2781] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:23:25.132293 30610 catalog_manager.cc:5495] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 reported cstate change: term changed from 1 to 2, leader changed from e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194) to 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193). New cstate: current_term: 2 leader_uuid: "9f55007ac96c440da32b0ef9e5e29281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:25.136366 30961 raft_consensus.cc:2454] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 5.
I20260805 22:23:25.137809 31200 raft_consensus.cc:686] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 5 LEADER]: Becoming Leader. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Running, Role: LEADER
I20260805 22:23:25.138599 31200 consensus_queue.cc:237] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:25.145128 30610 catalog_manager.cc:5495] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 reported cstate change: term changed from 4 to 5, leader changed from 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195) to 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193). New cstate: current_term: 5 leader_uuid: "9f55007ac96c440da32b0ef9e5e29281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:25.281097 31225 raft_consensus.cc:979] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4: : Instructing follower 9f55007ac96c440da32b0ef9e5e29281 to start an election
I20260805 22:23:25.281566 31241 raft_consensus.cc:1067] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 LEADER]: Signalling peer 9f55007ac96c440da32b0ef9e5e29281 to start an election
I20260805 22:23:25.282953 30764 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eaa4b450242b446fb1884b71a6bf55e5"
dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
from {username='slave'} at 127.0.0.1:54034
I20260805 22:23:25.283514 30764 raft_consensus.cc:482] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:23:25.283797 30764 raft_consensus.cc:3037] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:23:25.288553 30764 raft_consensus.cc:504] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:25.290328 30764 leader_election.cc:288] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 3 election: Requested vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:25.290958 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa4b450242b446fb1884b71a6bf55e5" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:25.291510 30860 raft_consensus.cc:3037] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:23:25.291692 30961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa4b450242b446fb1884b71a6bf55e5" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:25.292201 30961 raft_consensus.cc:3032] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:23:25.292434 30961 raft_consensus.cc:729] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Running, Role: LEADER
I20260805 22:23:25.293076 30961 consensus_queue.cc:260] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:25.294063 30961 raft_consensus.cc:3037] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:23:25.296269 30860 raft_consensus.cc:2454] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 3.
I20260805 22:23:25.297163 30709 leader_election.cc:302] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f55007ac96c440da32b0ef9e5e29281, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:25.297868 31200 raft_consensus.cc:2781] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:23:25.298290 31200 raft_consensus.cc:686] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Becoming Leader. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Running, Role: LEADER
I20260805 22:23:25.299335 30961 raft_consensus.cc:2454] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 3.
I20260805 22:23:25.299041 31200 consensus_queue.cc:237] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:25.306113 30610 catalog_manager.cc:5495] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 reported cstate change: term changed from 2 to 3, leader changed from 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195) to 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "9f55007ac96c440da32b0ef9e5e29281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:25.503959 31231 raft_consensus.cc:979] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3: : Instructing follower 9f55007ac96c440da32b0ef9e5e29281 to start an election
I20260805 22:23:25.504456 31232 raft_consensus.cc:1067] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 2 LEADER]: Signalling peer 9f55007ac96c440da32b0ef9e5e29281 to start an election
I20260805 22:23:25.505836 30764 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5"
dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
from {username='slave'} at 127.0.0.1:54030
I20260805 22:23:25.506325 30764 raft_consensus.cc:482] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:23:25.506614 30764 raft_consensus.cc:3037] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:23:25.511619 30764 raft_consensus.cc:504] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:25.513496 30764 leader_election.cc:288] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 3 election: Requested vote from peers e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:25.514173 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:25.514742 30860 raft_consensus.cc:3032] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:23:25.514953 30961 raft_consensus.cc:1261] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Refusing update from remote peer 9f55007ac96c440da32b0ef9e5e29281: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:23:25.515071 30860 raft_consensus.cc:729] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Running, Role: LEADER
I20260805 22:23:25.515463 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5" candidate_uuid: "9f55007ac96c440da32b0ef9e5e29281" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:25.515985 30963 raft_consensus.cc:3037] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:23:25.515915 30860 consensus_queue.cc:260] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:25.516898 31200 consensus_queue.cc:1035] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260805 22:23:25.517014 31232 consensus_queue.cc:1159] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "5366b554194f4acdb8916d0b1b2289e4" responder_term: 2 status { last_received { term: 2 index: 2 } last_committed_idx: 2 last_received_current_leader { term: 2 index: 2 } }
I20260805 22:23:25.518080 30860 raft_consensus.cc:3037] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:23:25.522708 30963 raft_consensus.cc:2454] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 3.
I20260805 22:23:25.524489 30860 raft_consensus.cc:2454] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f55007ac96c440da32b0ef9e5e29281 in term 3.
I20260805 22:23:25.525228 30716 leader_election.cc:302] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, 9f55007ac96c440da32b0ef9e5e29281; no voters:
I20260805 22:23:25.526942 31233 raft_consensus.cc:2781] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:23:25.528162 31233 raft_consensus.cc:686] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Becoming Leader. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Running, Role: LEADER
I20260805 22:23:25.528853 30860 raft_consensus.cc:1261] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Refusing update from remote peer 9f55007ac96c440da32b0ef9e5e29281: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:23:25.529196 31233 consensus_queue.cc:237] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:25.531111 31243 consensus_queue.cc:1035] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:23:25.544687 30610 catalog_manager.cc:5495] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 reported cstate change: term changed from 2 to 3, leader changed from e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194) to 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "9f55007ac96c440da32b0ef9e5e29281" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:25.564118 30860 raft_consensus.cc:1261] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Refusing update from remote peer 9f55007ac96c440da32b0ef9e5e29281: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:23:25.565358 31233 consensus_queue.cc:1035] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:25.574738 30962 raft_consensus.cc:1261] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Refusing update from remote peer 9f55007ac96c440da32b0ef9e5e29281: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:23:25.577207 31243 consensus_queue.cc:1035] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:25.614087 30860 raft_consensus.cc:1261] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 5 FOLLOWER]: Refusing update from remote peer 9f55007ac96c440da32b0ef9e5e29281: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260805 22:23:25.615270 31243 consensus_queue.cc:1035] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:25.623692 30962 raft_consensus.cc:1261] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 5 FOLLOWER]: Refusing update from remote peer 9f55007ac96c440da32b0ef9e5e29281: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260805 22:23:25.625548 31233 consensus_queue.cc:1035] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:25.724592 30860 raft_consensus.cc:1261] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Refusing update from remote peer 9f55007ac96c440da32b0ef9e5e29281: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:23:25.725936 31233 consensus_queue.cc:1035] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:25.734829 30962 raft_consensus.cc:1261] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Refusing update from remote peer 9f55007ac96c440da32b0ef9e5e29281: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:23:25.736271 31200 consensus_queue.cc:1035] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:25.995715 30860 raft_consensus.cc:1261] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Refusing update from remote peer 9f55007ac96c440da32b0ef9e5e29281: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:23:25.997539 31200 consensus_queue.cc:1035] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:23:26.011901 30962 raft_consensus.cc:1261] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Refusing update from remote peer 9f55007ac96c440da32b0ef9e5e29281: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:23:26.013280 31200 consensus_queue.cc:1035] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:28.054620 29783 auto_leader_rebalancer-test.cc:373] The leader distribution is
I20260805 22:23:28.055305 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:28.061987 30764 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5bbd9c17620b4dc1bfc9cd69188253e9"
dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
mode: GRACEFUL
new_leader_uuid: "e0691006952a44cfa5f2397f172bfcf3"
from {username='slave'} at 127.0.0.1:50950
I20260805 22:23:28.062605 30764 raft_consensus.cc:595] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 LEADER]: Received request to transfer leadership to TS e0691006952a44cfa5f2397f172bfcf3
I20260805 22:23:28.064986 30764 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "77a3f906013c4bf3bc7b79a0777c0213"
dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
mode: GRACEFUL
new_leader_uuid: "5366b554194f4acdb8916d0b1b2289e4"
from {username='slave'} at 127.0.0.1:50950
I20260805 22:23:28.065472 30764 raft_consensus.cc:595] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Received request to transfer leadership to TS 5366b554194f4acdb8916d0b1b2289e4
I20260805 22:23:28.067437 30764 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5"
dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
mode: GRACEFUL
new_leader_uuid: "5366b554194f4acdb8916d0b1b2289e4"
from {username='slave'} at 127.0.0.1:50950
I20260805 22:23:28.067881 30764 raft_consensus.cc:595] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Received request to transfer leadership to TS 5366b554194f4acdb8916d0b1b2289e4
I20260805 22:23:28.069911 30764 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eaa4b450242b446fb1884b71a6bf55e5"
dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
mode: GRACEFUL
new_leader_uuid: "e0691006952a44cfa5f2397f172bfcf3"
from {username='slave'} at 127.0.0.1:50950
I20260805 22:23:28.070348 30764 raft_consensus.cc:595] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Received request to transfer leadership to TS e0691006952a44cfa5f2397f172bfcf3
I20260805 22:23:28.071159 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260805 22:23:28.071633 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:28.134943 31274 raft_consensus.cc:979] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281: : Instructing follower 5366b554194f4acdb8916d0b1b2289e4 to start an election
I20260805 22:23:28.135496 31233 raft_consensus.cc:1067] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Signalling peer 5366b554194f4acdb8916d0b1b2289e4 to start an election
I20260805 22:23:28.137015 30963 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "77a3f906013c4bf3bc7b79a0777c0213"
dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
from {username='slave'} at 127.0.0.1:55454
I20260805 22:23:28.137578 30963 raft_consensus.cc:482] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:23:28.137925 30963 raft_consensus.cc:3037] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:23:28.144286 30963 raft_consensus.cc:504] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:28.146052 30963 leader_election.cc:288] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 4 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397)
I20260805 22:23:28.146797 30764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a3f906013c4bf3bc7b79a0777c0213" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:28.147085 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77a3f906013c4bf3bc7b79a0777c0213" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:28.147536 30764 raft_consensus.cc:3032] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:23:28.147552 30860 raft_consensus.cc:3037] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:23:28.147905 30764 raft_consensus.cc:729] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Running, Role: LEADER
I20260805 22:23:28.148630 30764 consensus_queue.cc:260] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:28.149633 30764 raft_consensus.cc:3037] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:23:28.152276 30860 raft_consensus.cc:2454] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 4.
I20260805 22:23:28.153330 30909 leader_election.cc:302] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:28.154136 31241 raft_consensus.cc:2781] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:23:28.154462 30764 raft_consensus.cc:2454] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 4.
I20260805 22:23:28.155166 31241 raft_consensus.cc:686] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 4 LEADER]: Becoming Leader. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Running, Role: LEADER
I20260805 22:23:28.156139 31241 consensus_queue.cc:237] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:28.162232 30609 catalog_manager.cc:5495] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 reported cstate change: term changed from 3 to 4, leader changed from 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193) to 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195). New cstate: current_term: 4 leader_uuid: "5366b554194f4acdb8916d0b1b2289e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:28.266166 31274 raft_consensus.cc:979] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281: : Instructing follower e0691006952a44cfa5f2397f172bfcf3 to start an election
I20260805 22:23:28.266851 31233 raft_consensus.cc:1067] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Signalling peer e0691006952a44cfa5f2397f172bfcf3 to start an election
I20260805 22:23:28.268263 30860 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eaa4b450242b446fb1884b71a6bf55e5"
dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
from {username='slave'} at 127.0.0.1:45188
I20260805 22:23:28.268718 30860 raft_consensus.cc:482] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:23:28.268992 30860 raft_consensus.cc:3037] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:23:28.273494 30860 raft_consensus.cc:504] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:28.275123 30860 leader_election.cc:288] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 4 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:28.276049 30764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa4b450242b446fb1884b71a6bf55e5" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:28.276283 30963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eaa4b450242b446fb1884b71a6bf55e5" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:28.276752 30963 raft_consensus.cc:3037] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:23:28.276678 30764 raft_consensus.cc:3032] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:23:28.277187 30764 raft_consensus.cc:729] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Running, Role: LEADER
I20260805 22:23:28.278012 30764 consensus_queue.cc:260] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:28.279244 30764 raft_consensus.cc:3037] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:23:28.283429 30963 raft_consensus.cc:2454] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 4.
I20260805 22:23:28.284745 30820 leader_election.cc:302] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:28.285488 30764 raft_consensus.cc:2454] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 4.
I20260805 22:23:28.285588 31273 raft_consensus.cc:2781] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:23:28.286160 31273 raft_consensus.cc:686] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 4 LEADER]: Becoming Leader. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Running, Role: LEADER
I20260805 22:23:28.286952 31273 consensus_queue.cc:237] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:28.294198 30609 catalog_manager.cc:5495] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 reported cstate change: term changed from 3 to 4, leader changed from 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193) to e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194). New cstate: current_term: 4 leader_uuid: "e0691006952a44cfa5f2397f172bfcf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:28.306660 31233 raft_consensus.cc:979] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281: : Instructing follower 5366b554194f4acdb8916d0b1b2289e4 to start an election
I20260805 22:23:28.307201 31285 raft_consensus.cc:1067] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Signalling peer 5366b554194f4acdb8916d0b1b2289e4 to start an election
I20260805 22:23:28.308596 30962 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5"
dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
from {username='slave'} at 127.0.0.1:55454
I20260805 22:23:28.309108 30962 raft_consensus.cc:482] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:23:28.309499 30962 raft_consensus.cc:3037] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:23:28.314419 30962 raft_consensus.cc:504] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:28.315905 30962 leader_election.cc:288] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 4 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194:37397)
I20260805 22:23:28.316740 30764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:28.316876 30860 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4407f4a7a884d60aeae1a2bd23e76d5" candidate_uuid: "5366b554194f4acdb8916d0b1b2289e4" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
I20260805 22:23:28.317363 30860 raft_consensus.cc:3037] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:23:28.317363 30764 raft_consensus.cc:3032] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:23:28.317850 30764 raft_consensus.cc:729] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Running, Role: LEADER
I20260805 22:23:28.318615 30764 consensus_queue.cc:260] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:28.319849 30764 raft_consensus.cc:3037] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:23:28.324026 30860 raft_consensus.cc:2454] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 4.
I20260805 22:23:28.325086 30909 leader_election.cc:302] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:28.325899 31241 raft_consensus.cc:2781] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:23:28.326267 31241 raft_consensus.cc:686] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 4 LEADER]: Becoming Leader. State: Replica: 5366b554194f4acdb8916d0b1b2289e4, State: Running, Role: LEADER
I20260805 22:23:28.326313 30764 raft_consensus.cc:2454] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5366b554194f4acdb8916d0b1b2289e4 in term 4.
I20260805 22:23:28.327073 31241 consensus_queue.cc:237] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:28.333967 30609 catalog_manager.cc:5495] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 reported cstate change: term changed from 3 to 4, leader changed from 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193) to 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195). New cstate: current_term: 4 leader_uuid: "5366b554194f4acdb8916d0b1b2289e4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:28.392373 31285 raft_consensus.cc:979] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281: : Instructing follower e0691006952a44cfa5f2397f172bfcf3 to start an election
I20260805 22:23:28.392787 31233 raft_consensus.cc:1067] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 LEADER]: Signalling peer e0691006952a44cfa5f2397f172bfcf3 to start an election
I20260805 22:23:28.394032 30860 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5bbd9c17620b4dc1bfc9cd69188253e9"
dest_uuid: "e0691006952a44cfa5f2397f172bfcf3"
from {username='slave'} at 127.0.0.1:45188
I20260805 22:23:28.394433 30860 raft_consensus.cc:482] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:23:28.394707 30860 raft_consensus.cc:3037] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:23:28.398957 30860 raft_consensus.cc:504] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:28.400251 30860 leader_election.cc:288] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 3 election: Requested vote from peers 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435), 5366b554194f4acdb8916d0b1b2289e4 (127.29.21.195:33339)
I20260805 22:23:28.401110 30764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbd9c17620b4dc1bfc9cd69188253e9" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "9f55007ac96c440da32b0ef9e5e29281"
I20260805 22:23:28.401257 30962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbd9c17620b4dc1bfc9cd69188253e9" candidate_uuid: "e0691006952a44cfa5f2397f172bfcf3" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "5366b554194f4acdb8916d0b1b2289e4"
I20260805 22:23:28.401600 30764 raft_consensus.cc:3032] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:23:28.401826 30962 raft_consensus.cc:3037] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:23:28.401948 30764 raft_consensus.cc:729] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 9f55007ac96c440da32b0ef9e5e29281, State: Running, Role: LEADER
I20260805 22:23:28.402665 30764 consensus_queue.cc:260] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:28.403688 30764 raft_consensus.cc:3037] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:23:28.406622 30962 raft_consensus.cc:2454] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 3.
I20260805 22:23:28.407473 30820 leader_election.cc:302] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5366b554194f4acdb8916d0b1b2289e4, e0691006952a44cfa5f2397f172bfcf3; no voters:
I20260805 22:23:28.407876 30764 raft_consensus.cc:2454] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0691006952a44cfa5f2397f172bfcf3 in term 3.
I20260805 22:23:28.408061 31273 raft_consensus.cc:2781] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:23:28.408535 31273 raft_consensus.cc:686] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 3 LEADER]: Becoming Leader. State: Replica: e0691006952a44cfa5f2397f172bfcf3, State: Running, Role: LEADER
I20260805 22:23:28.409271 31273 consensus_queue.cc:237] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } }
I20260805 22:23:28.415531 30609 catalog_manager.cc:5495] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 reported cstate change: term changed from 2 to 3, leader changed from 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193) to e0691006952a44cfa5f2397f172bfcf3 (127.29.21.194). New cstate: current_term: 3 leader_uuid: "e0691006952a44cfa5f2397f172bfcf3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:28.576658 30764 raft_consensus.cc:1261] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 4 FOLLOWER]: Refusing update from remote peer 5366b554194f4acdb8916d0b1b2289e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:23:28.577872 31241 consensus_queue.cc:1035] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:28.589556 30860 raft_consensus.cc:1261] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 4 FOLLOWER]: Refusing update from remote peer 5366b554194f4acdb8916d0b1b2289e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:23:28.591187 31277 consensus_queue.cc:1035] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:28.700088 30962 raft_consensus.cc:1261] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 4 FOLLOWER]: Refusing update from remote peer e0691006952a44cfa5f2397f172bfcf3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:23:28.701277 31273 consensus_queue.cc:1035] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:28.710213 30764 raft_consensus.cc:1261] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 4 FOLLOWER]: Refusing update from remote peer e0691006952a44cfa5f2397f172bfcf3: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:23:28.711324 31273 consensus_queue.cc:1035] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:28.769774 30860 raft_consensus.cc:1261] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 4 FOLLOWER]: Refusing update from remote peer 5366b554194f4acdb8916d0b1b2289e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:23:28.770861 31277 consensus_queue.cc:1035] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0691006952a44cfa5f2397f172bfcf3" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 37397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:28.779132 30764 raft_consensus.cc:1261] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 4 FOLLOWER]: Refusing update from remote peer 5366b554194f4acdb8916d0b1b2289e4: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:23:28.780438 31277 consensus_queue.cc:1035] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:28.868678 30764 raft_consensus.cc:1261] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Refusing update from remote peer e0691006952a44cfa5f2397f172bfcf3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:23:28.869823 31273 consensus_queue.cc:1035] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f55007ac96c440da32b0ef9e5e29281" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 37435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:28.879721 30962 raft_consensus.cc:1261] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Refusing update from remote peer e0691006952a44cfa5f2397f172bfcf3: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:23:28.881513 31308 consensus_queue.cc:1035] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5366b554194f4acdb8916d0b1b2289e4" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 33339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:23:29.072495 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:29.077646 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:29.078037 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:30.079026 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:30.085784 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:30.086217 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:31.087054 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:31.092299 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:31.092761 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:32.093680 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:32.099255 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:32.099704 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:33.100562 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:33.106247 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:33.106686 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:34.107446 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:34.112620 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:34.113042 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:35.113912 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:35.119216 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:35.119658 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:36.120812 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:36.128526 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:36.128957 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:37.129837 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:23:37.135073 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 0
I20260805 22:23:37.135529 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:23:38.164835 29783 auto_leader_rebalancer-test.cc:384] The leader distribution is
I20260805 22:23:38.165567 29783 tablet_server.cc:178] TabletServer@127.29.21.193:0 shutting down...
W20260805 22:23:38.169232 30916 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer [suppressed 4 similar messages]
W20260805 22:23:38.172968 30916 consensus_peers.cc:487] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 -> Peer 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435): Couldn't send request to peer 9f55007ac96c440da32b0ef9e5e29281. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:38.192193 29783 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260805 22:23:38.192914 29783 tablet_replica.cc:330] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281: stopping tablet replica
I20260805 22:23:38.193703 29783 raft_consensus.cc:2229] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:23:38.194907 29783 raft_consensus.cc:2258] T 0a6d6d5cabf34c998bba078b482c0b05 P 9f55007ac96c440da32b0ef9e5e29281 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.197144 29783 tablet_replica.cc:330] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281: stopping tablet replica
I20260805 22:23:38.197726 29783 raft_consensus.cc:2229] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.198184 29783 raft_consensus.cc:2258] T e9a7e5cf5aec44cf9cec9a2024024c5f P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.199918 29783 tablet_replica.cc:330] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281: stopping tablet replica
I20260805 22:23:38.200380 29783 raft_consensus.cc:2229] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.200793 29783 raft_consensus.cc:2258] T eaa4b450242b446fb1884b71a6bf55e5 P 9f55007ac96c440da32b0ef9e5e29281 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.202689 29783 tablet_replica.cc:330] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281: stopping tablet replica
I20260805 22:23:38.203276 29783 raft_consensus.cc:2229] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.203724 29783 raft_consensus.cc:2258] T 77a3f906013c4bf3bc7b79a0777c0213 P 9f55007ac96c440da32b0ef9e5e29281 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.205778 29783 tablet_replica.cc:330] T 8a62b2e2dc4145aca8e0a45586dfe57d P 9f55007ac96c440da32b0ef9e5e29281: stopping tablet replica
I20260805 22:23:38.206322 29783 raft_consensus.cc:2229] T 8a62b2e2dc4145aca8e0a45586dfe57d P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.206691 29783 raft_consensus.cc:2258] T 8a62b2e2dc4145aca8e0a45586dfe57d P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.208442 29783 tablet_replica.cc:330] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281: stopping tablet replica
I20260805 22:23:38.208869 29783 raft_consensus.cc:2229] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 LEADER]: Raft consensus shutting down.
I20260805 22:23:38.209690 29783 raft_consensus.cc:2258] T de85cd15b9354ebbbd8be638aa855495 P 9f55007ac96c440da32b0ef9e5e29281 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.211301 29783 tablet_replica.cc:330] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281: stopping tablet replica
I20260805 22:23:38.211756 29783 raft_consensus.cc:2229] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.212141 29783 raft_consensus.cc:2258] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.213829 29783 tablet_replica.cc:330] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281: stopping tablet replica
I20260805 22:23:38.214254 29783 raft_consensus.cc:2229] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.214617 29783 raft_consensus.cc:2258] T a4407f4a7a884d60aeae1a2bd23e76d5 P 9f55007ac96c440da32b0ef9e5e29281 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.216120 29783 tablet_replica.cc:330] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281: stopping tablet replica
I20260805 22:23:38.216514 29783 raft_consensus.cc:2229] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 5 LEADER]: Raft consensus shutting down.
I20260805 22:23:38.217345 29783 raft_consensus.cc:2258] T 7960edb18a3a4203b8c683499c8fe92c P 9f55007ac96c440da32b0ef9e5e29281 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.219141 29783 tablet_replica.cc:330] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281: stopping tablet replica
I20260805 22:23:38.219524 29783 raft_consensus.cc:2229] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 3 LEADER]: Raft consensus shutting down.
I20260805 22:23:38.220314 29783 raft_consensus.cc:2258] T bd2a35893d484eae80fe9e16ba237fcc P 9f55007ac96c440da32b0ef9e5e29281 [term 3 FOLLOWER]: Raft consensus is shut down!
W20260805 22:23:38.235810 30916 consensus_peers.cc:487] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 -> Peer 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435): Couldn't send request to peer 9f55007ac96c440da32b0ef9e5e29281. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on TabletServer. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:38.250686 29783 tablet_server.cc:194] TabletServer@127.29.21.193:0 shutdown complete.
W20260805 22:23:38.271716 30916 consensus_peers.cc:487] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 -> Peer 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435): Couldn't send request to peer 9f55007ac96c440da32b0ef9e5e29281. Status: Network error: Client connection negotiation failed: client connection to 127.29.21.193:37435: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:38.272954 29783 tablet_server.cc:178] TabletServer@127.29.21.194:0 shutting down...
I20260805 22:23:38.301954 29783 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260805 22:23:38.302881 29783 tablet_replica.cc:330] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3: stopping tablet replica
I20260805 22:23:38.303745 29783 raft_consensus.cc:2229] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260805 22:23:38.304006 30820 consensus_peers.cc:487] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 -> Peer 9f55007ac96c440da32b0ef9e5e29281 (127.29.21.193:37435): Couldn't send request to peer 9f55007ac96c440da32b0ef9e5e29281. Status: Network error: Client connection negotiation failed: client connection to 127.29.21.193:37435: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:38.304339 29783 raft_consensus.cc:2258] T 0a6d6d5cabf34c998bba078b482c0b05 P e0691006952a44cfa5f2397f172bfcf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.306193 29783 tablet_replica.cc:330] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3: stopping tablet replica
I20260805 22:23:38.306676 29783 raft_consensus.cc:2229] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 3 LEADER]: Raft consensus shutting down.
I20260805 22:23:38.307392 29783 raft_consensus.cc:2258] T 5bbd9c17620b4dc1bfc9cd69188253e9 P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.308980 29783 tablet_replica.cc:330] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3: stopping tablet replica
I20260805 22:23:38.309508 29783 raft_consensus.cc:2229] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.309937 29783 raft_consensus.cc:2258] T de85cd15b9354ebbbd8be638aa855495 P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.311606 29783 tablet_replica.cc:330] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3: stopping tablet replica
I20260805 22:23:38.312040 29783 raft_consensus.cc:2229] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.312395 29783 raft_consensus.cc:2258] T a4407f4a7a884d60aeae1a2bd23e76d5 P e0691006952a44cfa5f2397f172bfcf3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.314069 29783 tablet_replica.cc:330] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3: stopping tablet replica
I20260805 22:23:38.314510 29783 raft_consensus.cc:2229] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.314909 29783 raft_consensus.cc:2258] T 7960edb18a3a4203b8c683499c8fe92c P e0691006952a44cfa5f2397f172bfcf3 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.316381 29783 tablet_replica.cc:330] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3: stopping tablet replica
I20260805 22:23:38.316823 29783 raft_consensus.cc:2229] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.317163 29783 raft_consensus.cc:2258] T bd2a35893d484eae80fe9e16ba237fcc P e0691006952a44cfa5f2397f172bfcf3 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.318951 29783 tablet_replica.cc:330] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3: stopping tablet replica
I20260805 22:23:38.319382 29783 raft_consensus.cc:2229] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.319736 29783 raft_consensus.cc:2258] T 8a62b2e2dc4145aca8e0a45586dfe57d P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.321195 29783 tablet_replica.cc:330] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3: stopping tablet replica
I20260805 22:23:38.321656 29783 raft_consensus.cc:2229] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 4 LEADER]: Raft consensus shutting down.
I20260805 22:23:38.322398 29783 raft_consensus.cc:2258] T eaa4b450242b446fb1884b71a6bf55e5 P e0691006952a44cfa5f2397f172bfcf3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.324046 29783 tablet_replica.cc:330] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3: stopping tablet replica
I20260805 22:23:38.324493 29783 raft_consensus.cc:2229] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.324891 29783 raft_consensus.cc:2258] T 77a3f906013c4bf3bc7b79a0777c0213 P e0691006952a44cfa5f2397f172bfcf3 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.326431 29783 tablet_replica.cc:330] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3: stopping tablet replica
I20260805 22:23:38.326835 29783 raft_consensus.cc:2229] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [term 2 LEADER]: Raft consensus shutting down.
I20260805 22:23:38.327569 29783 raft_consensus.cc:2258] T e9a7e5cf5aec44cf9cec9a2024024c5f P e0691006952a44cfa5f2397f172bfcf3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.374094 29783 tablet_server.cc:194] TabletServer@127.29.21.194:0 shutdown complete.
I20260805 22:23:38.394582 29783 tablet_server.cc:178] TabletServer@127.29.21.195:0 shutting down...
I20260805 22:23:38.424662 29783 ts_tablet_manager.cc:1500] Shutting down tablet manager...
I20260805 22:23:38.425340 29783 tablet_replica.cc:330] T 0a6d6d5cabf34c998bba078b482c0b05 P 5366b554194f4acdb8916d0b1b2289e4: stopping tablet replica
I20260805 22:23:38.426007 29783 raft_consensus.cc:2229] T 0a6d6d5cabf34c998bba078b482c0b05 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.426484 29783 raft_consensus.cc:2258] T 0a6d6d5cabf34c998bba078b482c0b05 P 5366b554194f4acdb8916d0b1b2289e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.428226 29783 tablet_replica.cc:330] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4: stopping tablet replica
I20260805 22:23:38.428717 29783 raft_consensus.cc:2229] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [term 2 LEADER]: Raft consensus shutting down.
I20260805 22:23:38.429529 29783 raft_consensus.cc:2258] T 8a62b2e2dc4145aca8e0a45586dfe57d P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.431272 29783 tablet_replica.cc:330] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4: stopping tablet replica
I20260805 22:23:38.431758 29783 raft_consensus.cc:2229] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.432178 29783 raft_consensus.cc:2258] T bd2a35893d484eae80fe9e16ba237fcc P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.433898 29783 tablet_replica.cc:330] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4: stopping tablet replica
I20260805 22:23:38.434403 29783 raft_consensus.cc:2229] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 5 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.434789 29783 raft_consensus.cc:2258] T 7960edb18a3a4203b8c683499c8fe92c P 5366b554194f4acdb8916d0b1b2289e4 [term 5 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.436329 29783 tablet_replica.cc:330] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4: stopping tablet replica
I20260805 22:23:38.436789 29783 raft_consensus.cc:2229] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 4 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.437204 29783 raft_consensus.cc:2258] T eaa4b450242b446fb1884b71a6bf55e5 P 5366b554194f4acdb8916d0b1b2289e4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.438772 29783 tablet_replica.cc:330] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4: stopping tablet replica
I20260805 22:23:38.439218 29783 raft_consensus.cc:2229] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.439597 29783 raft_consensus.cc:2258] T e9a7e5cf5aec44cf9cec9a2024024c5f P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.441044 29783 tablet_replica.cc:330] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4: stopping tablet replica
I20260805 22:23:38.441522 29783 raft_consensus.cc:2229] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 4 LEADER]: Raft consensus shutting down.
I20260805 22:23:38.442317 29783 raft_consensus.cc:2258] T 77a3f906013c4bf3bc7b79a0777c0213 P 5366b554194f4acdb8916d0b1b2289e4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.443847 29783 tablet_replica.cc:330] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4: stopping tablet replica
I20260805 22:23:38.444264 29783 raft_consensus.cc:2229] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.444662 29783 raft_consensus.cc:2258] T 5bbd9c17620b4dc1bfc9cd69188253e9 P 5366b554194f4acdb8916d0b1b2289e4 [term 3 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.446259 29783 tablet_replica.cc:330] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4: stopping tablet replica
I20260805 22:23:38.446668 29783 raft_consensus.cc:2229] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:38.447046 29783 raft_consensus.cc:2258] T de85cd15b9354ebbbd8be638aa855495 P 5366b554194f4acdb8916d0b1b2289e4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.448521 29783 tablet_replica.cc:330] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4: stopping tablet replica
I20260805 22:23:38.448930 29783 raft_consensus.cc:2229] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 4 LEADER]: Raft consensus shutting down.
I20260805 22:23:38.449708 29783 raft_consensus.cc:2258] T a4407f4a7a884d60aeae1a2bd23e76d5 P 5366b554194f4acdb8916d0b1b2289e4 [term 4 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.478554 29783 tablet_server.cc:194] TabletServer@127.29.21.195:0 shutdown complete.
I20260805 22:23:38.499320 29783 master.cc:537] Master@127.29.21.254:39533 shutting down...
I20260805 22:23:38.526664 29783 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [term 1 LEADER]: Raft consensus shutting down.
I20260805 22:23:38.527235 29783 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:38.527545 29783 tablet_replica.cc:330] T 00000000000000000000000000000000 P 154e2d921d3d4e94b77b95c4a741385c: stopping tablet replica
I20260805 22:23:38.546718 29783 master.cc:552] Master@127.29.21.254:39533 shutdown complete.
5366b554194f4acdb8916d0b1b2289e4 1
9f55007ac96c440da32b0ef9e5e29281 5
e0691006952a44cfa5f2397f172bfcf3 4
5366b554194f4acdb8916d0b1b2289e4 3
9f55007ac96c440da32b0ef9e5e29281 3
e0691006952a44cfa5f2397f172bfcf3 4
5366b554194f4acdb8916d0b1b2289e4 1
9f55007ac96c440da32b0ef9e5e29281 8
e0691006952a44cfa5f2397f172bfcf3 1
5366b554194f4acdb8916d0b1b2289e4 3
9f55007ac96c440da32b0ef9e5e29281 4
e0691006952a44cfa5f2397f172bfcf3 3
[ OK ] LeaderRebalancerTest.FunctionalTestForNotDivided (30213 ms)
[ RUN ] LeaderRebalancerTest.AddTserver
I20260805 22:23:38.586199 29783 internal_mini_cluster.cc:156] Creating distributed mini masters. Addrs: 127.29.21.254:43401
I20260805 22:23:38.587303 29783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:23:38.593446 31330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:23:38.593468 31329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:23:38.593689 31333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:23:38.595253 29783 server_base.cc:1034] running on GCE node
I20260805 22:23:38.596334 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260805 22:23:38.596513 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260805 22:23:38.596654 29783 hybrid_clock.cc:648] HybridClock initialized: now 1785968618596640 us; error 0 us; skew 500 ppm
I20260805 22:23:38.597172 29783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:23:38.604365 29783 webserver.cc:460] Webserver started at http://127.29.21.254:45215/ using document root <none> and password file <none>
I20260805 22:23:38.604842 29783 fs_manager.cc:359] Metadata directory not provided
I20260805 22:23:38.605011 29783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:23:38.605311 29783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:23:38.606592 29783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/master-0-root/instance:
uuid: "28a5659dcb9347329aa3accbe95f66b0"
format_stamp: "Formatted at 2026-08-05 22:23:38 on dist-test-slave-f0tn"
I20260805 22:23:38.611091 29783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 22:23:38.614431 31338 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:38.615128 29783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260805 22:23:38.615402 29783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/master-0-root
uuid: "28a5659dcb9347329aa3accbe95f66b0"
format_stamp: "Formatted at 2026-08-05 22:23:38 on dist-test-slave-f0tn"
I20260805 22:23:38.615666 29783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/master-0-root
metadata directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/master-0-root
1 data directories: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/master-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:23:38.630398 29783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:23:38.631657 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:23:38.671648 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.254:43401
I20260805 22:23:38.671752 31402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.21.254:43401 every 8 connection(s)
I20260805 22:23:38.675806 31403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:38.685930 31403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0: Bootstrap starting.
I20260805 22:23:38.690583 31403 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:38.694783 31403 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0: No bootstrap required, opened a new log
I20260805 22:23:38.696821 31403 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28a5659dcb9347329aa3accbe95f66b0" member_type: VOTER }
I20260805 22:23:38.697304 31403 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:38.697522 31403 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28a5659dcb9347329aa3accbe95f66b0, State: Initialized, Role: FOLLOWER
I20260805 22:23:38.698086 31403 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28a5659dcb9347329aa3accbe95f66b0" member_type: VOTER }
I20260805 22:23:38.698674 31403 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260805 22:23:38.698930 31403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260805 22:23:38.699155 31403 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:38.703150 31403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28a5659dcb9347329aa3accbe95f66b0" member_type: VOTER }
I20260805 22:23:38.703604 31403 leader_election.cc:302] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28a5659dcb9347329aa3accbe95f66b0; no voters:
I20260805 22:23:38.704659 31403 leader_election.cc:288] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260805 22:23:38.704964 31407 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:38.706282 31407 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0 [term 1 LEADER]: Becoming Leader. State: Replica: 28a5659dcb9347329aa3accbe95f66b0, State: Running, Role: LEADER
I20260805 22:23:38.706939 31407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28a5659dcb9347329aa3accbe95f66b0" member_type: VOTER }
I20260805 22:23:38.707538 31403 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260805 22:23:38.711759 31408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28a5659dcb9347329aa3accbe95f66b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28a5659dcb9347329aa3accbe95f66b0" member_type: VOTER } }
I20260805 22:23:38.711934 31409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28a5659dcb9347329aa3accbe95f66b0. Latest consensus state: current_term: 1 leader_uuid: "28a5659dcb9347329aa3accbe95f66b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28a5659dcb9347329aa3accbe95f66b0" member_type: VOTER } }
I20260805 22:23:38.712549 31409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0 [sys.catalog]: This master's current role is: LEADER
I20260805 22:23:38.712553 31408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0 [sys.catalog]: This master's current role is: LEADER
I20260805 22:23:38.719167 31418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260805 22:23:38.726140 31418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260805 22:23:38.727169 29783 internal_mini_cluster.cc:184] Waiting to initialize catalog manager on master 0
I20260805 22:23:38.734961 31418 catalog_manager.cc:1347] Generated new cluster ID: c34c36d561d1418bb75c4d2b44f0bfb8
I20260805 22:23:38.735316 31418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260805 22:23:38.752816 31418 catalog_manager.cc:1370] Generated new certificate authority record
I20260805 22:23:38.754499 31418 catalog_manager.cc:1504] Loading token signing keys...
I20260805 22:23:38.766206 31418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0: Generated new TSK 0
I20260805 22:23:38.766886 31418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260805 22:23:38.794385 29783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:23:38.800565 31433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:23:38.801483 31434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:23:38.803308 31436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:23:38.803614 29783 server_base.cc:1034] running on GCE node
I20260805 22:23:38.804682 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260805 22:23:38.804867 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260805 22:23:38.805032 29783 hybrid_clock.cc:648] HybridClock initialized: now 1785968618805018 us; error 0 us; skew 500 ppm
I20260805 22:23:38.805622 29783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:23:38.808032 29783 webserver.cc:460] Webserver started at http://127.29.21.193:39829/ using document root <none> and password file <none>
I20260805 22:23:38.808503 29783 fs_manager.cc:359] Metadata directory not provided
I20260805 22:23:38.808683 29783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:23:38.808928 29783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:23:38.810179 29783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-0-root/instance:
uuid: "f8e9f4c756da4203917c464e2b179994"
format_stamp: "Formatted at 2026-08-05 22:23:38 on dist-test-slave-f0tn"
I20260805 22:23:38.814746 29783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260805 22:23:38.818190 31443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:38.818877 29783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 22:23:38.819162 29783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-0-root
uuid: "f8e9f4c756da4203917c464e2b179994"
format_stamp: "Formatted at 2026-08-05 22:23:38 on dist-test-slave-f0tn"
I20260805 22:23:38.819442 29783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-0-root
metadata directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-0-root
1 data directories: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-0-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:23:38.856288 29783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:23:38.857599 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:23:38.859098 29783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:23:38.861346 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:23:38.861554 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:38.861802 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:23:38.861958 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:38.901839 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.193:45607
I20260805 22:23:38.901950 31535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.21.193:45607 every 8 connection(s)
I20260805 22:23:38.906656 29783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:23:38.913301 31541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:23:38.914103 31542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:23:38.921103 29783 server_base.cc:1034] running on GCE node
W20260805 22:23:38.922812 31544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:23:38.923509 31536 heartbeater.cc:347] Connected to a master server at 127.29.21.254:43401
I20260805 22:23:38.923828 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260805 22:23:38.923935 31536 heartbeater.cc:464] Registering TS with master...
W20260805 22:23:38.924121 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260805 22:23:38.924312 29783 hybrid_clock.cc:648] HybridClock initialized: now 1785968618924297 us; error 0 us; skew 500 ppm
I20260805 22:23:38.924804 31536 heartbeater.cc:511] Master 127.29.21.254:43401 requested a full tablet report, sending...
I20260805 22:23:38.925113 29783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:23:38.927201 31359 ts_manager.cc:194] Registered new tserver with Master: f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:23:38.928148 29783 webserver.cc:460] Webserver started at http://127.29.21.194:43217/ using document root <none> and password file <none>
I20260805 22:23:38.928791 29783 fs_manager.cc:359] Metadata directory not provided
I20260805 22:23:38.929039 29783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:23:38.929347 29783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:23:38.929966 31359 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34388
I20260805 22:23:38.930967 29783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-1-root/instance:
uuid: "516cb07ce3064ccbbca35e912d62816c"
format_stamp: "Formatted at 2026-08-05 22:23:38 on dist-test-slave-f0tn"
I20260805 22:23:38.935624 29783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260805 22:23:38.938977 31552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:38.939797 29783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260805 22:23:38.940116 29783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-1-root
uuid: "516cb07ce3064ccbbca35e912d62816c"
format_stamp: "Formatted at 2026-08-05 22:23:38 on dist-test-slave-f0tn"
I20260805 22:23:38.940388 29783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-1-root
metadata directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-1-root
1 data directories: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-1-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:23:38.951586 29783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:23:38.952754 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:23:38.954247 29783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:23:38.956328 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:23:38.956506 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:38.956720 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:23:38.956857 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:38.995200 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.194:41897
I20260805 22:23:38.995296 31643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.21.194:41897 every 8 connection(s)
I20260805 22:23:38.999792 29783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:23:39.009040 31655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:23:39.009735 31653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:23:39.009938 31658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:23:39.012696 29783 server_base.cc:1034] running on GCE node
I20260805 22:23:39.013705 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
I20260805 22:23:39.013816 31646 heartbeater.cc:347] Connected to a master server at 127.29.21.254:43401
W20260805 22:23:39.013968 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260805 22:23:39.014182 29783 hybrid_clock.cc:648] HybridClock initialized: now 1785968619014164 us; error 0 us; skew 500 ppm
I20260805 22:23:39.014223 31646 heartbeater.cc:464] Registering TS with master...
I20260805 22:23:39.014962 29783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:23:39.015096 31646 heartbeater.cc:511] Master 127.29.21.254:43401 requested a full tablet report, sending...
I20260805 22:23:39.017318 31359 ts_manager.cc:194] Registered new tserver with Master: 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:39.018086 29783 webserver.cc:460] Webserver started at http://127.29.21.195:40251/ using document root <none> and password file <none>
I20260805 22:23:39.018725 29783 fs_manager.cc:359] Metadata directory not provided
I20260805 22:23:39.018958 29783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:23:39.019275 29783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:23:39.019518 31359 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34394
I20260805 22:23:39.020804 29783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-2-root/instance:
uuid: "feb5e7ab4b1541beaf2674c3c5500385"
format_stamp: "Formatted at 2026-08-05 22:23:39 on dist-test-slave-f0tn"
I20260805 22:23:39.025913 29783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260805 22:23:39.029052 31666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:39.029789 29783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260805 22:23:39.030050 29783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-2-root
uuid: "feb5e7ab4b1541beaf2674c3c5500385"
format_stamp: "Formatted at 2026-08-05 22:23:39 on dist-test-slave-f0tn"
I20260805 22:23:39.030308 29783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-2-root
metadata directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-2-root
1 data directories: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-2-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:23:39.040366 29783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:23:39.041679 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:23:39.043207 29783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:23:39.045357 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:23:39.045629 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:39.045850 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:23:39.045990 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:39.084014 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.195:44831
I20260805 22:23:39.084111 31748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.21.195:44831 every 8 connection(s)
I20260805 22:23:39.099224 31749 heartbeater.cc:347] Connected to a master server at 127.29.21.254:43401
I20260805 22:23:39.099607 31749 heartbeater.cc:464] Registering TS with master...
I20260805 22:23:39.100298 31749 heartbeater.cc:511] Master 127.29.21.254:43401 requested a full tablet report, sending...
I20260805 22:23:39.102190 31359 ts_manager.cc:194] Registered new tserver with Master: feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:39.102810 29783 internal_mini_cluster.cc:371] 3 TS(s) registered with all masters after 0.015886738s
I20260805 22:23:39.103724 31359 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:34400
I20260805 22:23:39.105113 29783 test_util.cc:268] Using random seed: 958344000
I20260805 22:23:39.129835 31359 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34408:
name: "test-workload"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}""\004\001\000\313c+\002\004\001\000\226\307V\004\004\001\000a+\202\006\004\001\000,\217\255\010\004\001\000\367\362\330\n\004\001\000\302V\004\r\004\001\000\215\272/\017\004\001\000X\036[\021\004\001\000#\202\206\023\004\001\000\356\345\261\025\004\001\000\271I\335\027\004\001\000\204\255\010\032\004\001\000O\0214\034\004\001\000\032u_\036\004\001\000\345\330\212 \004\001\000\260<\266\"\004\001\000{\240\341$\004\001\000F\004\r\'\004\001\000\021h8)\004\001\000\334\313c+\004\001\000\247/\217-\004\001\000r\223\272/\004\001\000=\367\3451\004\001\000\010[\0214\004\001\000\323\276<6\004\001\000\236\"h8\004\001\000i\206\223:\004\001\0004\352\276<\004\001\000\377M\352>\004\001\000\312\261\025A\004\001\000\225\025AC\004\001\000`ylE\004\001\000+\335\227G\004\001\000\366@\303I\004\001\000\301\244\356K\004\001\000\214\010\032N\004\001\000WlEP\004\001\000\"\320pR\004\001\000\3553\234T\004\001\000\270\227\307V\004\001\000\203\373\362X\004\001\000N_\036[\004\001\000\031\303I]\004\001\000\344&u_\004\001\000\257\212\240a\004\001\000z\356\313c\004\001\000ER\367e\004\001\000\020\266\"h\004\001\000\333\031Nj\004\001\000\246}yl\004\001\000q\341\244n\004\001\000<E\320p\004\001\000\007\251\373r\004\001\000\322\014\'u\004\001\000\235pRw\004\001\000h\324}y\004\001\00038\251{\004\001\000\376\233\324}"
indirect_data: """"
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260805 22:23:39.135794 31359 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-workload in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260805 22:23:39.335788 31592 tablet_service.cc:1467] Processing CreateTablet for tablet c46ca6ab004a4f2c90ecccced3235cb6 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260805 22:23:39.336450 31589 tablet_service.cc:1467] Processing CreateTablet for tablet 32fee80a830a450195ce9c3462460ca1 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260805 22:23:39.337242 31592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46ca6ab004a4f2c90ecccced3235cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.338587 31587 tablet_service.cc:1467] Processing CreateTablet for tablet 4172338662524cb29310f48e0861873e (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260805 22:23:39.339664 31586 tablet_service.cc:1467] Processing CreateTablet for tablet 39e7b32cb2f646828a02f900ec2d47f1 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260805 22:23:39.340201 31587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4172338662524cb29310f48e0861873e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.340976 31586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e7b32cb2f646828a02f900ec2d47f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.338546 31590 tablet_service.cc:1467] Processing CreateTablet for tablet 3095cdb25afc4c26bd9d1e03a919b474 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260805 22:23:39.341614 31580 tablet_service.cc:1467] Processing CreateTablet for tablet aee974c178e84c3d9a5dcfb31556a46f (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260805 22:23:39.342724 31590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3095cdb25afc4c26bd9d1e03a919b474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.343820 31581 tablet_service.cc:1467] Processing CreateTablet for tablet 3f76fdd524184682ae27abc293a7fff6 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260805 22:23:39.345072 31581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f76fdd524184682ae27abc293a7fff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.346518 31582 tablet_service.cc:1467] Processing CreateTablet for tablet 45cd56d4e96040f98dbf1ae001a1005a (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260805 22:23:39.347798 31582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cd56d4e96040f98dbf1ae001a1005a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.350498 31583 tablet_service.cc:1467] Processing CreateTablet for tablet 49bb7cd9b51b4dd782a68b560d7570f2 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260805 22:23:39.351810 31583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49bb7cd9b51b4dd782a68b560d7570f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.354517 31585 tablet_service.cc:1467] Processing CreateTablet for tablet 58a08ace2d684c14972d495f4a387a85 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260805 22:23:39.356045 31585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58a08ace2d684c14972d495f4a387a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.359169 31589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32fee80a830a450195ce9c3462460ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.361847 31580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aee974c178e84c3d9a5dcfb31556a46f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.363519 31586 tablet_service.cc:1467] Processing CreateTablet for tablet c5b0fd7f6d7a46b2a77e841ceda358ef (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260805 22:23:39.364734 31586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b0fd7f6d7a46b2a77e841ceda358ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.373713 31590 tablet_service.cc:1467] Processing CreateTablet for tablet 94d0d26905694943b7135a9b805cf106 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260805 22:23:39.375085 31590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d0d26905694943b7135a9b805cf106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.379506 31586 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7039038a134310977029d3b3fb9fc9 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260805 22:23:39.380878 31586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7039038a134310977029d3b3fb9fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.386628 31768 tablet_bootstrap.cc:492] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:39.387158 31587 tablet_service.cc:1467] Processing CreateTablet for tablet 0191f03fa8984a40afc7ec1e44970c8e (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260805 22:23:39.388543 31587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0191f03fa8984a40afc7ec1e44970c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.369030 31592 tablet_service.cc:1467] Processing CreateTablet for tablet d264e8379ab54b4785c836a00397ba95 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260805 22:23:39.390209 31592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d264e8379ab54b4785c836a00397ba95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.399533 31485 tablet_service.cc:1467] Processing CreateTablet for tablet c46ca6ab004a4f2c90ecccced3235cb6 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260805 22:23:39.400789 31768 tablet_bootstrap.cc:654] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:39.400928 31485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46ca6ab004a4f2c90ecccced3235cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.401890 31586 tablet_service.cc:1467] Processing CreateTablet for tablet ab66683a9e5e46c2b09be4afdee2a08c (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260805 22:23:39.403137 31586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab66683a9e5e46c2b09be4afdee2a08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.404927 31587 tablet_service.cc:1467] Processing CreateTablet for tablet 89a7aa19faed455fb2600ae6851aee0e (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260805 22:23:39.406227 31587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89a7aa19faed455fb2600ae6851aee0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.407361 31484 tablet_service.cc:1467] Processing CreateTablet for tablet 3095cdb25afc4c26bd9d1e03a919b474 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260805 22:23:39.408674 31484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3095cdb25afc4c26bd9d1e03a919b474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.415189 31483 tablet_service.cc:1467] Processing CreateTablet for tablet 32fee80a830a450195ce9c3462460ca1 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260805 22:23:39.416656 31483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32fee80a830a450195ce9c3462460ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.422062 31482 tablet_service.cc:1467] Processing CreateTablet for tablet 4172338662524cb29310f48e0861873e (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260805 22:23:39.423962 31482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4172338662524cb29310f48e0861873e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.425824 31481 tablet_service.cc:1467] Processing CreateTablet for tablet 39e7b32cb2f646828a02f900ec2d47f1 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260805 22:23:39.426219 31478 tablet_service.cc:1467] Processing CreateTablet for tablet 49bb7cd9b51b4dd782a68b560d7570f2 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260805 22:23:39.427176 31481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e7b32cb2f646828a02f900ec2d47f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.428272 31476 tablet_service.cc:1467] Processing CreateTablet for tablet 45cd56d4e96040f98dbf1ae001a1005a (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260805 22:23:39.429553 31476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cd56d4e96040f98dbf1ae001a1005a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.429322 31586 tablet_service.cc:1467] Processing CreateTablet for tablet 5e4ad7072aab4331878e988a26a6d887 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260805 22:23:39.430007 31479 tablet_service.cc:1467] Processing CreateTablet for tablet 58a08ace2d684c14972d495f4a387a85 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260805 22:23:39.430583 31586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e4ad7072aab4331878e988a26a6d887. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.431234 31479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58a08ace2d684c14972d495f4a387a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.431596 31473 tablet_service.cc:1467] Processing CreateTablet for tablet 3f76fdd524184682ae27abc293a7fff6 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260805 22:23:39.438764 31473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f76fdd524184682ae27abc293a7fff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.448717 31472 tablet_service.cc:1467] Processing CreateTablet for tablet aee974c178e84c3d9a5dcfb31556a46f (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260805 22:23:39.463094 31478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49bb7cd9b51b4dd782a68b560d7570f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.464692 31472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aee974c178e84c3d9a5dcfb31556a46f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.464998 31587 tablet_service.cc:1467] Processing CreateTablet for tablet 8abfbdc4afff482ebc81c4fda5d0812a (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260805 22:23:39.466328 31587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8abfbdc4afff482ebc81c4fda5d0812a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.466156 31590 tablet_service.cc:1467] Processing CreateTablet for tablet 07310455e35e48f8ba4eb9b72a69113b (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260805 22:23:39.467470 31590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07310455e35e48f8ba4eb9b72a69113b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.483479 31583 tablet_service.cc:1467] Processing CreateTablet for tablet dc38ff59fd654f6689e3a88fdbb765cf (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260805 22:23:39.484489 31583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc38ff59fd654f6689e3a88fdbb765cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.506917 31585 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff925b5d86b41c19d809f76e57ab062 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260805 22:23:39.510188 31585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff925b5d86b41c19d809f76e57ab062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.519249 31768 tablet_bootstrap.cc:492] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:39.519750 31768 ts_tablet_manager.cc:1397] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.133s user 0.005s sys 0.023s
I20260805 22:23:39.522629 31768 raft_consensus.cc:348] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:39.523517 31768 raft_consensus.cc:374] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:39.523883 31768 raft_consensus.cc:729] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:39.526705 31580 tablet_service.cc:1467] Processing CreateTablet for tablet 4af8570c6c58453685d7a8f38b519905 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260805 22:23:39.527158 31590 tablet_service.cc:1467] Processing CreateTablet for tablet 52af4d94f396434d997b46f52763cd20 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260805 22:23:39.531219 31582 tablet_service.cc:1467] Processing CreateTablet for tablet eda619941ce64f788d0b7a364eed0bdb (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260805 22:23:39.532289 31483 tablet_service.cc:1467] Processing CreateTablet for tablet d264e8379ab54b4785c836a00397ba95 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260805 22:23:39.532526 31590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52af4d94f396434d997b46f52763cd20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.527158 31485 tablet_service.cc:1467] Processing CreateTablet for tablet 94d0d26905694943b7135a9b805cf106 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260805 22:23:39.534317 31580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4af8570c6c58453685d7a8f38b519905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.529340 31587 tablet_service.cc:1467] Processing CreateTablet for tablet 5df9cd03cde746e0956c8c7726d4eb70 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260805 22:23:39.534969 31582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eda619941ce64f788d0b7a364eed0bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.529265 31583 tablet_service.cc:1467] Processing CreateTablet for tablet 9d68a856424b4ceb8f4f54b4050e8dc4 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260805 22:23:39.535620 31473 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7039038a134310977029d3b3fb9fc9 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260805 22:23:39.536114 31583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d68a856424b4ceb8f4f54b4050e8dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.539278 31484 tablet_service.cc:1467] Processing CreateTablet for tablet 0191f03fa8984a40afc7ec1e44970c8e (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260805 22:23:39.540203 31587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5df9cd03cde746e0956c8c7726d4eb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.532632 31586 tablet_service.cc:1467] Processing CreateTablet for tablet c43b2a80b8c446bdb2ac0e9ca04e8cf4 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260805 22:23:39.534562 31771 tablet_bootstrap.cc:492] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:39.542193 31586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43b2a80b8c446bdb2ac0e9ca04e8cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.525185 31768 consensus_queue.cc:260] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:39.545953 31581 tablet_service.cc:1467] Processing CreateTablet for tablet a260d4eadd0a4677aed088f331367c05 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260805 22:23:39.547205 31581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a260d4eadd0a4677aed088f331367c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.534739 31485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d0d26905694943b7135a9b805cf106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.553740 31473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7039038a134310977029d3b3fb9fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.557272 31589 tablet_service.cc:1467] Processing CreateTablet for tablet 4deaa72a885a43bfbc6171a7ed13ad81 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260805 22:23:39.558848 31589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4deaa72a885a43bfbc6171a7ed13ad81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.565753 31484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0191f03fa8984a40afc7ec1e44970c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:39.512599 31472 tablet_service.cc:1467] Processing CreateTablet for tablet c5b0fd7f6d7a46b2a77e841ceda358ef (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260805 22:23:39.546066 31481 tablet_service.cc:1467] Processing CreateTablet for tablet ab66683a9e5e46c2b09be4afdee2a08c (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260805 22:23:39.566563 31479 tablet_service.cc:1467] Processing CreateTablet for tablet 89a7aa19faed455fb2600ae6851aee0e (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260805 22:23:39.607036 31592 tablet_service.cc:1467] Processing CreateTablet for tablet 6f372cf6a3ec4a4cac2b6335a1a64a56 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260805 22:23:39.988747 31536 heartbeater.cc:503] Master 127.29.21.254:43401 was elected leader, sending a full tablet report...
W20260805 22:23:40.211813 31647 tablet.cc:2369] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260805 22:23:40.107157 31749 heartbeater.cc:503] Master 127.29.21.254:43401 was elected leader, sending a full tablet report...
I20260805 22:23:39.620450 31768 ts_tablet_manager.cc:1428] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.100s user 0.012s sys 0.001s
W20260805 22:23:40.396996 31768 ts_tablet_manager.cc:1460] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c: Tablet startup took 876ms
W20260805 22:23:40.397408 31768 ts_tablet_manager.cc:1462] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c: Trace:
0805 22:23:39.335715 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:39.337612 (+ 1897us) service_pool.cc:227] Handling call
0805 22:23:39.339495 (+ 1883us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:39.339597 (+ 102us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:39.348728 (+ 9131us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:39.354957 (+ 6229us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:39.385952 (+ 30995us) inbound_call.cc:173] Queueing success response
0805 22:23:39.386146 (+ 194us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0805 22:23:39.520032 (+133886us) ts_tablet_manager.cc:1429] Starting tablet replica
0805 22:23:39.520519 (+ 487us) tablet_replica.cc:259] Starting instrumentation
0805 22:23:39.521111 (+ 592us) tablet_replica.cc:278] Starting consensus
Metrics: {"raft.queue_time_us":87032,"raft.run_cpu_time_us":183,"raft.run_wall_time_us":185,"tablet-open.queue_time_us":30761,"thread_start_us":103509,"threads_started":2}
I20260805 22:23:40.400377 31768 tablet_bootstrap.cc:492] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:40.400192 31582 tablet_service.cc:1467] Processing CreateTablet for tablet 18a7e330200a49349edcae379f1e2b57 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260805 22:23:40.359508 31646 heartbeater.cc:503] Master 127.29.21.254:43401 was elected leader, sending a full tablet report...
I20260805 22:23:40.401629 31589 tablet_service.cc:1467] Processing CreateTablet for tablet 10de517cc0a24f20b5885091e99e252d (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260805 22:23:40.360445 31483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d264e8379ab54b4785c836a00397ba95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.405007 31478 tablet_service.cc:1467] Processing CreateTablet for tablet 5e4ad7072aab4331878e988a26a6d887 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260805 22:23:40.405251 31585 tablet_service.cc:1467] Processing CreateTablet for tablet 78cb483d17014b3281316012dbb6f6e1 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260805 22:23:40.407140 31478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e4ad7072aab4331878e988a26a6d887. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.408381 31473 tablet_service.cc:1467] Processing CreateTablet for tablet 8abfbdc4afff482ebc81c4fda5d0812a (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260805 22:23:40.408794 31581 tablet_service.cc:1467] Processing CreateTablet for tablet 527d60ab6db74595bed7e87ca2ecc6e8 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260805 22:23:40.409619 31473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8abfbdc4afff482ebc81c4fda5d0812a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.412382 31583 tablet_service.cc:1467] Processing CreateTablet for tablet 6b16a2e0814140e39609da87479bb57e (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
W20260805 22:23:40.362119 31423 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260805 22:23:40.363381 31774 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:40.413918 31774 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:40.417024 31484 tablet_service.cc:1467] Processing CreateTablet for tablet dc38ff59fd654f6689e3a88fdbb765cf (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260805 22:23:40.418352 31484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc38ff59fd654f6689e3a88fdbb765cf. 1 dirs total, 0 dirs full, 0 dirs failed
W20260805 22:23:40.372877 31476 consensus_meta.cc:287] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.785s user 0.004s sys 0.013s
I20260805 22:23:40.377617 31592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f372cf6a3ec4a4cac2b6335a1a64a56. 1 dirs total, 0 dirs full, 0 dirs failed
W20260805 22:23:40.384598 31485 consensus_meta.cc:287] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.819s user 0.018s sys 0.019s
I20260805 22:23:40.422358 31580 tablet_service.cc:1467] Processing CreateTablet for tablet dcea14dac2934d139e2c5ab6d0b8648d (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260805 22:23:40.433564 31702 tablet_service.cc:1467] Processing CreateTablet for tablet 3095cdb25afc4c26bd9d1e03a919b474 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 36398027 <= VALUES < 72796054
I20260805 22:23:40.434639 31580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcea14dac2934d139e2c5ab6d0b8648d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.434806 31702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3095cdb25afc4c26bd9d1e03a919b474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.434742 31485 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 12) took 1006 ms. Trace:
I20260805 22:23:40.435242 31485 rpcz_store.cc:276] 0805 22:23:39.428301 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:39.526340 (+ 98039us) service_pool.cc:227] Handling call
0805 22:23:39.534295 (+ 7955us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:39.534391 (+ 96us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:39.564821 (+ 30430us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.434201 (+869380us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.434685 (+ 484us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:23:40.437076 31485 tablet_service.cc:1467] Processing CreateTablet for tablet 07310455e35e48f8ba4eb9b72a69113b (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260805 22:23:40.438073 31583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b16a2e0814140e39609da87479bb57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.423092 31771 tablet_bootstrap.cc:654] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:40.425698 31472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b0fd7f6d7a46b2a77e841ceda358ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.425310 31587 tablet_service.cc:1467] Processing CreateTablet for tablet 986c28960d44411487f79ca134af59e2 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260805 22:23:40.441762 31585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78cb483d17014b3281316012dbb6f6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.442341 31587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986c28960d44411487f79ca134af59e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.442898 31485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07310455e35e48f8ba4eb9b72a69113b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.445261 31581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527d60ab6db74595bed7e87ca2ecc6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.445665 31479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89a7aa19faed455fb2600ae6851aee0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.429656 31482 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 3) took 1008 ms. Trace:
I20260805 22:23:40.431311 31703 tablet_service.cc:1467] Processing CreateTablet for tablet c46ca6ab004a4f2c90ecccced3235cb6 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION VALUES < 36398027
I20260805 22:23:40.449527 31482 rpcz_store.cc:276] 0805 22:23:39.420923 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:39.421060 (+ 137us) service_pool.cc:227] Handling call
0805 22:23:39.423370 (+ 2310us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:39.423484 (+ 114us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:39.540733 (+117249us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.031149 (+490416us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.429593 (+398444us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:23:40.450202 31703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c46ca6ab004a4f2c90ecccced3235cb6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.451354 31482 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff925b5d86b41c19d809f76e57ab062 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260805 22:23:40.452560 31482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff925b5d86b41c19d809f76e57ab062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.453851 31589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10de517cc0a24f20b5885091e99e252d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.457708 31582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a7e330200a49349edcae379f1e2b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.432746 31774 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:40.427977 31590 tablet_service.cc:1467] Processing CreateTablet for tablet c75e772a0f924080865ddf85ed1e2d86 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260805 22:23:40.462497 31590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c75e772a0f924080865ddf85ed1e2d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.462082 31700 tablet_service.cc:1467] Processing CreateTablet for tablet 32fee80a830a450195ce9c3462460ca1 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 72796054 <= VALUES < 109194081
I20260805 22:23:40.464987 31700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32fee80a830a450195ce9c3462460ca1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.464691 31698 tablet_service.cc:1467] Processing CreateTablet for tablet 4172338662524cb29310f48e0861873e (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 109194081 <= VALUES < 145592108
I20260805 22:23:40.465996 31698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4172338662524cb29310f48e0861873e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.470181 31476 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 7) took 1044 ms. Trace:
I20260805 22:23:40.470657 31476 rpcz_store.cc:276] 0805 22:23:39.425635 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:39.427291 (+ 1656us) service_pool.cc:227] Handling call
0805 22:23:39.429025 (+ 1734us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:39.429115 (+ 90us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:39.573857 (+144742us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.469635 (+895778us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.470117 (+ 482us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:23:40.472620 31476 tablet_service.cc:1467] Processing CreateTablet for tablet 52af4d94f396434d997b46f52763cd20 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260805 22:23:40.477002 31697 tablet_service.cc:1467] Processing CreateTablet for tablet 39e7b32cb2f646828a02f900ec2d47f1 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 145592108 <= VALUES < 181990135
I20260805 22:23:40.486233 31697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e7b32cb2f646828a02f900ec2d47f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.481721 31696 tablet_service.cc:1467] Processing CreateTablet for tablet 58a08ace2d684c14972d495f4a387a85 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 181990135 <= VALUES < 218388162
I20260805 22:23:40.484073 31695 tablet_service.cc:1467] Processing CreateTablet for tablet 49bb7cd9b51b4dd782a68b560d7570f2 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 218388162 <= VALUES < 254786189
I20260805 22:23:40.486379 31694 tablet_service.cc:1467] Processing CreateTablet for tablet 45cd56d4e96040f98dbf1ae001a1005a (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 254786189 <= VALUES < 291184216
I20260805 22:23:40.488430 31692 tablet_service.cc:1467] Processing CreateTablet for tablet aee974c178e84c3d9a5dcfb31556a46f (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 327582243 <= VALUES < 363980270
I20260805 22:23:40.488552 31693 tablet_service.cc:1467] Processing CreateTablet for tablet 3f76fdd524184682ae27abc293a7fff6 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 291184216 <= VALUES < 327582243
I20260805 22:23:40.490087 31694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45cd56d4e96040f98dbf1ae001a1005a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.494293 31696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58a08ace2d684c14972d495f4a387a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.494933 31693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f76fdd524184682ae27abc293a7fff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.495473 31692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aee974c178e84c3d9a5dcfb31556a46f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.501768 31695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49bb7cd9b51b4dd782a68b560d7570f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.503640 31771 tablet_bootstrap.cc:492] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:40.504122 31771 ts_tablet_manager.cc:1397] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.970s user 0.090s sys 0.006s
I20260805 22:23:40.514261 31586 tablet_service.cc:1467] Processing CreateTablet for tablet fe370453251f4300a8d90c397021595c (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260805 22:23:40.515681 31586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe370453251f4300a8d90c397021595c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.521757 31481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab66683a9e5e46c2b09be4afdee2a08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.526497 31590 tablet_service.cc:1467] Processing CreateTablet for tablet 1115c6a30f64413ab24c9d215d58a2c7 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260805 22:23:40.527812 31590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1115c6a30f64413ab24c9d215d58a2c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.534749 31700 tablet_service.cc:1467] Processing CreateTablet for tablet 5e4ad7072aab4331878e988a26a6d887 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 618766459 <= VALUES < 655164486
I20260805 22:23:40.536068 31700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e4ad7072aab4331878e988a26a6d887. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.537994 31703 tablet_service.cc:1467] Processing CreateTablet for tablet c5b0fd7f6d7a46b2a77e841ceda358ef (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 363980270 <= VALUES < 400378297
I20260805 22:23:40.538758 31702 tablet_service.cc:1467] Processing CreateTablet for tablet d264e8379ab54b4785c836a00397ba95 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 400378297 <= VALUES < 436776324
I20260805 22:23:40.539336 31703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5b0fd7f6d7a46b2a77e841ceda358ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.539999 31702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d264e8379ab54b4785c836a00397ba95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.541303 31587 tablet_service.cc:1467] Processing CreateTablet for tablet b185bcb30c384758983895f8e5d8c98a (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260805 22:23:40.546700 31587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b185bcb30c384758983895f8e5d8c98a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.563006 31768 tablet_bootstrap.cc:654] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:40.563958 31472 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 10) took 1136 ms. Trace:
I20260805 22:23:40.564458 31472 rpcz_store.cc:276] 0805 22:23:39.427301 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:39.511607 (+ 84306us) service_pool.cc:227] Handling call
0805 22:23:40.425117 (+913510us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:40.425252 (+ 135us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.448862 (+ 23610us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.563377 (+114515us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.563889 (+ 512us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10624}
I20260805 22:23:40.569789 31476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52af4d94f396434d997b46f52763cd20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.577801 31694 tablet_service.cc:1467] Processing CreateTablet for tablet 94d0d26905694943b7135a9b805cf106 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 436776324 <= VALUES < 473174351
I20260805 22:23:40.579223 31473 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 18) took 1147 ms. Trace:
I20260805 22:23:40.582468 31473 rpcz_store.cc:276] 0805 22:23:39.432295 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:40.407546 (+975251us) service_pool.cc:227] Handling call
0805 22:23:40.409167 (+ 1621us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:40.409256 (+ 89us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.478905 (+ 69649us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.578653 (+ 99748us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.579161 (+ 508us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:23:40.584606 31473 tablet_service.cc:1467] Processing CreateTablet for tablet c43b2a80b8c446bdb2ac0e9ca04e8cf4 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
I20260805 22:23:40.586133 31473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43b2a80b8c446bdb2ac0e9ca04e8cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.587404 31483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 11) took 1159 ms. Trace:
I20260805 22:23:40.587872 31483 rpcz_store.cc:276] 0805 22:23:39.427811 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:39.525525 (+ 97714us) service_pool.cc:227] Handling call
0805 22:23:39.542670 (+ 17145us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:39.542828 (+ 158us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.412433 (+869605us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.586845 (+174412us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.587337 (+ 492us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:23:40.579304 31484 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 19) took 1135 ms. Trace:
I20260805 22:23:40.602574 31484 rpcz_store.cc:276] 0805 22:23:39.447299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:40.416146 (+968847us) service_pool.cc:227] Handling call
0805 22:23:40.417870 (+ 1724us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:40.417968 (+ 98us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.476028 (+ 58060us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.578786 (+102758us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.579246 (+ 460us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:23:40.606467 31484 tablet_service.cc:1467] Processing CreateTablet for tablet 4af8570c6c58453685d7a8f38b519905 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260805 22:23:40.608227 31484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4af8570c6c58453685d7a8f38b519905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.612782 31478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 17) took 1181 ms. Trace:
I20260805 22:23:40.614418 31694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d0d26905694943b7135a9b805cf106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.634035 31696 tablet_service.cc:1467] Processing CreateTablet for tablet 2d7039038a134310977029d3b3fb9fc9 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 473174351 <= VALUES < 509572378
I20260805 22:23:40.638234 31472 tablet_service.cc:1467] Processing CreateTablet for tablet 5df9cd03cde746e0956c8c7726d4eb70 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260805 22:23:40.638849 31696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7039038a134310977029d3b3fb9fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.639632 31472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5df9cd03cde746e0956c8c7726d4eb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.639480 31692 tablet_service.cc:1467] Processing CreateTablet for tablet 89a7aa19faed455fb2600ae6851aee0e (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 582368432 <= VALUES < 618766459
I20260805 22:23:40.640910 31692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89a7aa19faed455fb2600ae6851aee0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.621223 31693 tablet_service.cc:1467] Processing CreateTablet for tablet 0191f03fa8984a40afc7ec1e44970c8e (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 509572378 <= VALUES < 545970405
I20260805 22:23:40.643128 31779 tablet_bootstrap.cc:492] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:40.629078 31697 tablet_service.cc:1467] Processing CreateTablet for tablet ab66683a9e5e46c2b09be4afdee2a08c (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 545970405 <= VALUES < 582368432
I20260805 22:23:40.649679 31585 tablet_service.cc:1467] Processing CreateTablet for tablet cd72177fb2f24347ba19da011ccd3fe4 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260805 22:23:40.653573 31592 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:59024 (request call id 30) took 1105 ms. Trace:
I20260805 22:23:40.647691 31479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 16) took 1216 ms. Trace:
I20260805 22:23:40.654256 31479 rpcz_store.cc:276] 0805 22:23:39.431552 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:39.565705 (+134153us) service_pool.cc:227] Handling call
0805 22:23:40.433464 (+867759us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:40.433601 (+ 137us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.546601 (+113000us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.647160 (+100559us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.647626 (+ 466us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:23:40.656836 31479 tablet_service.cc:1467] Processing CreateTablet for tablet eda619941ce64f788d0b7a364eed0bdb (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260805 22:23:40.655403 31585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd72177fb2f24347ba19da011ccd3fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.621337 31483 tablet_service.cc:1467] Processing CreateTablet for tablet 9d68a856424b4ceb8f4f54b4050e8dc4 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260805 22:23:40.654582 31771 raft_consensus.cc:348] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:40.660032 31771 raft_consensus.cc:374] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:40.660454 31771 raft_consensus.cc:729] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:40.661350 31771 consensus_queue.cc:260] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:40.660542 31483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d68a856424b4ceb8f4f54b4050e8dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.670423 31482 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 21) took 1173 ms. Trace:
I20260805 22:23:40.670864 31482 rpcz_store.cc:276] 0805 22:23:39.496773 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:40.450474 (+953701us) service_pool.cc:227] Handling call
0805 22:23:40.452140 (+ 1666us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:40.452229 (+ 89us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.587232 (+135003us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.665292 (+ 78060us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.670355 (+ 5063us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:23:40.672765 31482 tablet_service.cc:1467] Processing CreateTablet for tablet a260d4eadd0a4677aed088f331367c05 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260805 22:23:40.613215 31478 rpcz_store.cc:276] 0805 22:23:39.431956 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:40.404139 (+972183us) service_pool.cc:227] Handling call
0805 22:23:40.406593 (+ 2454us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:40.406705 (+ 112us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.543771 (+137066us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.612231 (+ 68460us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.612722 (+ 491us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:23:40.685724 31478 tablet_service.cc:1467] Processing CreateTablet for tablet 4deaa72a885a43bfbc6171a7ed13ad81 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260805 22:23:40.687572 31476 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 24) took 1159 ms. Trace:
I20260805 22:23:40.687950 31476 rpcz_store.cc:276] 0805 22:23:39.528434 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:40.471705 (+943271us) service_pool.cc:227] Handling call
0805 22:23:40.493984 (+ 22279us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:40.556687 (+ 62703us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.666992 (+110305us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.687073 (+ 20081us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.687512 (+ 439us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:23:40.674186 31479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eda619941ce64f788d0b7a364eed0bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.676891 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:40.677634 31698 tablet_service.cc:1467] Processing CreateTablet for tablet 8abfbdc4afff482ebc81c4fda5d0812a (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 655164486 <= VALUES < 691562513
I20260805 22:23:40.630136 31586 tablet_service.cc:1467] Processing CreateTablet for tablet 36ee93639c3f4084b94145c6ee60a2a8 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260805 22:23:40.653972 31592 rpcz_store.cc:276] 0805 22:23:39.548444 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:39.588651 (+ 40207us) service_pool.cc:227] Handling call
0805 22:23:40.377073 (+788422us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:40.377176 (+ 103us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.462031 (+ 84855us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.626319 (+164288us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.653504 (+ 27185us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3648}
I20260805 22:23:40.679804 31485 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 20) took 1225 ms. Trace:
I20260805 22:23:40.692870 31590 tablet_service.cc:1467] Processing CreateTablet for tablet 42324773ce144eb784e14693eecbee72 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260805 22:23:40.694391 31586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36ee93639c3f4084b94145c6ee60a2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.697036 31580 tablet_service.cc:1467] Processing CreateTablet for tablet 16e20d6ca2f34a65a5d7bc4471282feb (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260805 22:23:40.695756 31592 tablet_service.cc:1467] Processing CreateTablet for tablet f914c8243e724eda849ce1fc2dc30827 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260805 22:23:40.698661 31580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16e20d6ca2f34a65a5d7bc4471282feb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.696979 31485 rpcz_store.cc:276] 0805 22:23:39.454705 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:40.436228 (+981523us) service_pool.cc:227] Handling call
0805 22:23:40.438010 (+ 1782us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:40.438109 (+ 99us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.484755 (+ 46646us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.649297 (+164542us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.679739 (+ 30442us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:23:40.701583 31485 tablet_service.cc:1467] Processing CreateTablet for tablet fe370453251f4300a8d90c397021595c (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260805 22:23:40.704859 31581 tablet_service.cc:1467] Processing CreateTablet for tablet fb8dc633587f48d1bba2a54a1619bc3e (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260805 22:23:40.705271 31693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0191f03fa8984a40afc7ec1e44970c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.706494 31590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42324773ce144eb784e14693eecbee72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.707335 31592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f914c8243e724eda849ce1fc2dc30827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.712177 31779 tablet_bootstrap.cc:654] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:40.704870 31482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a260d4eadd0a4677aed088f331367c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.713789 31697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab66683a9e5e46c2b09be4afdee2a08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.714408 31485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe370453251f4300a8d90c397021595c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.719117 31581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8dc633587f48d1bba2a54a1619bc3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.721302 31481 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 15) took 1290 ms. Trace:
I20260805 22:23:40.721762 31481 rpcz_store.cc:276] 0805 22:23:39.431222 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:39.545300 (+114078us) service_pool.cc:227] Handling call
0805 22:23:39.588083 (+ 42783us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:39.981342 (+393259us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.599503 (+618161us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.720672 (+121169us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.721240 (+ 568us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3968}
I20260805 22:23:40.723450 31768 tablet_bootstrap.cc:492] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:40.723961 31768 ts_tablet_manager.cc:1397] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.324s user 0.024s sys 0.038s
I20260805 22:23:40.723683 31481 tablet_service.cc:1467] Processing CreateTablet for tablet 18a7e330200a49349edcae379f1e2b57 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260805 22:23:40.689150 31583 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1aa2dc5e5341f0a0450361d6c95814 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260805 22:23:40.727401 31583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1aa2dc5e5341f0a0450361d6c95814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.730639 31582 tablet_service.cc:1467] Processing CreateTablet for tablet f8863c96c162438983f403d38b11c5db (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260805 22:23:40.731032 31700 tablet_service.cc:1467] Processing CreateTablet for tablet dc38ff59fd654f6689e3a88fdbb765cf (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 691562513 <= VALUES < 727960540
I20260805 22:23:40.732419 31700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc38ff59fd654f6689e3a88fdbb765cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.733603 31478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4deaa72a885a43bfbc6171a7ed13ad81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.695753 31589 tablet_service.cc:1467] Processing CreateTablet for tablet 8513af2d574d45b4a6d02428543b1402 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260805 22:23:40.747146 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:40.748847 31554 leader_election.cc:341] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:40.750013 31476 tablet_service.cc:1467] Processing CreateTablet for tablet 6f372cf6a3ec4a4cac2b6335a1a64a56 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260805 22:23:40.751273 31476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f372cf6a3ec4a4cac2b6335a1a64a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.759119 31695 tablet_service.cc:1467] Processing CreateTablet for tablet c43b2a80b8c446bdb2ac0e9ca04e8cf4 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 800756594 <= VALUES < 837154621
W20260805 22:23:40.760123 31554 leader_election.cc:341] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:40.760172 31481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a7e330200a49349edcae379f1e2b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.760645 31554 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:40.761458 31774 raft_consensus.cc:2726] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:40.764990 31694 tablet_service.cc:1467] Processing CreateTablet for tablet 07310455e35e48f8ba4eb9b72a69113b (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 727960540 <= VALUES < 764358567
I20260805 22:23:40.774669 31482 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 28) took 1230 ms. Trace:
I20260805 22:23:40.775069 31482 rpcz_store.cc:276] 0805 22:23:39.544676 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:40.671827 (+1127151us) service_pool.cc:227] Handling call
0805 22:23:40.680892 (+ 9065us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:40.681003 (+ 111us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.747689 (+ 66686us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.774199 (+ 26510us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.774612 (+ 413us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:23:40.780457 31592 tablet_service.cc:1467] Processing CreateTablet for tablet d8fd526bfd0047e49803dbf11f8eafc6 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260805 22:23:40.766839 31768 raft_consensus.cc:348] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:40.745080 31473 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 22) took 1220 ms. Trace:
I20260805 22:23:40.770594 31771 ts_tablet_manager.cc:1428] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.266s user 0.008s sys 0.005s
I20260805 22:23:40.781751 31473 rpcz_store.cc:276] 0805 22:23:39.524899 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:40.583648 (+1058749us) service_pool.cc:227] Handling call
0805 22:23:40.585542 (+ 1894us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:40.585704 (+ 162us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.708416 (+122712us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.744503 (+ 36087us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.745004 (+ 501us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:23:40.782539 31771 tablet_bootstrap.cc:492] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:40.783877 31473 tablet_service.cc:1467] Processing CreateTablet for tablet 78cb483d17014b3281316012dbb6f6e1 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260805 22:23:40.785154 31473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78cb483d17014b3281316012dbb6f6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.788225 31771 tablet_bootstrap.cc:654] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:40.789903 31698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8abfbdc4afff482ebc81c4fda5d0812a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.781412 31768 raft_consensus.cc:374] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:40.791735 31768 raft_consensus.cc:729] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:40.792466 31695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43b2a80b8c446bdb2ac0e9ca04e8cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.773833 31587 tablet_service.cc:1467] Processing CreateTablet for tablet 7c99e6a491454b5ba8d31e3246e77797 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260805 22:23:40.797260 31694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07310455e35e48f8ba4eb9b72a69113b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.776840 31586 tablet_service.cc:1467] Processing CreateTablet for tablet 17b3240dd1b947178dafdc9cac7f1e5e (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260805 22:23:40.800052 31592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8fd526bfd0047e49803dbf11f8eafc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.800773 31589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8513af2d574d45b4a6d02428543b1402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.802127 31483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 25) took 1273 ms. Trace:
I20260805 22:23:40.792521 31768 consensus_queue.cc:260] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:40.802131 31587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c99e6a491454b5ba8d31e3246e77797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.805821 31582 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8863c96c162438983f403d38b11c5db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.806560 31586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17b3240dd1b947178dafdc9cac7f1e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.802546 31483 rpcz_store.cc:276] 0805 22:23:39.528772 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:40.620571 (+1091799us) service_pool.cc:227] Handling call
0805 22:23:40.659988 (+ 39417us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:40.660107 (+ 119us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.771213 (+111106us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.801509 (+ 30296us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.802060 (+ 551us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:23:40.814174 31483 tablet_service.cc:1467] Processing CreateTablet for tablet 527d60ab6db74595bed7e87ca2ecc6e8 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260805 22:23:40.819144 31768 ts_tablet_manager.cc:1428] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.095s user 0.009s sys 0.000s
I20260805 22:23:40.820281 31768 tablet_bootstrap.cc:492] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:40.815448 31483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527d60ab6db74595bed7e87ca2ecc6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.737165 31702 tablet_service.cc:1467] Processing CreateTablet for tablet 0ff925b5d86b41c19d809f76e57ab062 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 764358567 <= VALUES < 800756594
I20260805 22:23:40.835916 31702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ff925b5d86b41c19d809f76e57ab062. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.846614 31484 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 23) took 1321 ms. Trace:
I20260805 22:23:40.847047 31484 rpcz_store.cc:276] 0805 22:23:39.525334 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:40.604729 (+1079395us) service_pool.cc:227] Handling call
0805 22:23:40.607632 (+ 2903us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:40.607744 (+ 112us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.752343 (+144599us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.846120 (+ 93777us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.846550 (+ 430us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:23:40.849159 31484 tablet_service.cc:1467] Processing CreateTablet for tablet 6b16a2e0814140e39609da87479bb57e (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260805 22:23:40.776800 31482 tablet_service.cc:1467] Processing CreateTablet for tablet 10de517cc0a24f20b5885091e99e252d (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260805 22:23:40.849653 31581 tablet_service.cc:1467] Processing CreateTablet for tablet 9f41c2ff98f049c3bc654aa98cd77694 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260805 22:23:40.850477 31484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b16a2e0814140e39609da87479bb57e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.851087 31581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f41c2ff98f049c3bc654aa98cd77694. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.854763 31580 tablet_service.cc:1467] Processing CreateTablet for tablet bd1c7254b2b74c10a4fb473ea633544d (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260805 22:23:40.855986 31580 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1c7254b2b74c10a4fb473ea633544d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.859459 31779 tablet_bootstrap.cc:492] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:40.860026 31779 ts_tablet_manager.cc:1397] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.217s user 0.007s sys 0.009s
I20260805 22:23:40.860646 31583 tablet_service.cc:1467] Processing CreateTablet for tablet 91a9acdbc98a4e799374fe6e8950c3a7 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260805 22:23:40.862684 31779 raft_consensus.cc:348] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:40.863432 31779 raft_consensus.cc:374] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:40.863757 31482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10de517cc0a24f20b5885091e99e252d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.859444 31771 tablet_bootstrap.cc:492] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:40.865813 31771 ts_tablet_manager.cc:1397] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.083s user 0.011s sys 0.005s
I20260805 22:23:40.868520 31771 raft_consensus.cc:348] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:40.869957 31583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a9acdbc98a4e799374fe6e8950c3a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.870126 31768 tablet_bootstrap.cc:654] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:40.869239 31771 raft_consensus.cc:374] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:40.870579 31700 tablet_service.cc:1467] Processing CreateTablet for tablet 4af8570c6c58453685d7a8f38b519905 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 837154621 <= VALUES < 873552648
I20260805 22:23:40.871155 31771 raft_consensus.cc:729] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:40.872056 31700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4af8570c6c58453685d7a8f38b519905. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.872062 31771 consensus_queue.cc:260] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:40.861227 31478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 29) took 1295 ms. Trace:
I20260805 22:23:40.878465 31478 rpcz_store.cc:276] 0805 22:23:39.565702 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:40.684785 (+1119083us) service_pool.cc:227] Handling call
0805 22:23:40.686592 (+ 1807us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:40.686693 (+ 101us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.826042 (+139349us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.860738 (+ 34696us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.861169 (+ 431us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:23:40.880730 31478 tablet_service.cc:1467] Processing CreateTablet for tablet dcea14dac2934d139e2c5ab6d0b8648d (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260805 22:23:40.882146 31478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcea14dac2934d139e2c5ab6d0b8648d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260805 22:23:40.886457 31339 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 45294.8us
I20260805 22:23:40.890174 31697 tablet_service.cc:1467] Processing CreateTablet for tablet 9d68a856424b4ceb8f4f54b4050e8dc4 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 909950675 <= VALUES < 946348702
I20260805 22:23:40.892385 31786 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:40.895605 31786 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:40.874058 31703 tablet_service.cc:1467] Processing CreateTablet for tablet 52af4d94f396434d997b46f52763cd20 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 873552648 <= VALUES < 909950675
I20260805 22:23:40.898242 31786 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:40.898825 31771 ts_tablet_manager.cc:1428] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20260805 22:23:40.911423 31586 tablet_service.cc:1467] Processing CreateTablet for tablet e6cf90ae3fc840ceb48be8c790be97d9 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260805 22:23:40.912777 31586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6cf90ae3fc840ceb48be8c790be97d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.863827 31779 raft_consensus.cc:729] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:40.921834 31771 tablet_bootstrap.cc:492] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:40.922732 31476 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 30) took 1356 ms. Trace:
I20260805 22:23:40.923691 31479 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 27) took 1394 ms. Trace:
I20260805 22:23:40.925082 31481 tablet_service.cc:1467] Processing CreateTablet for tablet 986c28960d44411487f79ca134af59e2 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260805 22:23:40.932446 31703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 52af4d94f396434d997b46f52763cd20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.947302 31472 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:40714 (request call id 26) took 1418 ms. Trace:
I20260805 22:23:40.946039 31481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986c28960d44411487f79ca134af59e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.947732 31472 rpcz_store.cc:276] 0805 22:23:39.529068 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:40.630933 (+1101865us) service_pool.cc:227] Handling call
0805 22:23:40.639134 (+ 8201us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:40.639237 (+ 103us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.752733 (+113496us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.942519 (+189786us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.947235 (+ 4716us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:23:40.924943 31692 tablet_service.cc:1467] Processing CreateTablet for tablet 18a7e330200a49349edcae379f1e2b57 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1164736864 <= VALUES < 1201134891
I20260805 22:23:40.950383 31483 tablet_service.cc:1467] Processing CreateTablet for tablet c75e772a0f924080865ddf85ed1e2d86 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260805 22:23:40.955196 31483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c75e772a0f924080865ddf85ed1e2d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.958146 31694 tablet_service.cc:1467] Processing CreateTablet for tablet 5df9cd03cde746e0956c8c7726d4eb70 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 946348702 <= VALUES < 982746729
I20260805 22:23:40.966344 31768 tablet_bootstrap.cc:492] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:40.966873 31768 ts_tablet_manager.cc:1397] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.147s user 0.018s sys 0.000s
I20260805 22:23:40.967382 31771 tablet_bootstrap.cc:654] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:40.949456 31479 rpcz_store.cc:276] 0805 22:23:39.529418 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:40.655935 (+1126517us) service_pool.cc:227] Handling call
0805 22:23:40.672091 (+ 16156us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:40.672477 (+ 386us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.864867 (+192390us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.923247 (+ 58380us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.923636 (+ 389us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12255}
I20260805 22:23:40.972028 31479 tablet_service.cc:1467] Processing CreateTablet for tablet b185bcb30c384758983895f8e5d8c98a (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260805 22:23:40.973313 31479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b185bcb30c384758983895f8e5d8c98a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.943905 31476 rpcz_store.cc:276] 0805 22:23:39.566015 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:23:40.688865 (+1122850us) service_pool.cc:227] Handling call
0805 22:23:40.750820 (+ 61955us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0805 22:23:40.750910 (+ 90us) ts_tablet_manager.cc:676] Creating new metadata...
0805 22:23:40.782598 (+ 31688us) tablet_metadata.cc:641] Metadata flushed
0805 22:23:40.922207 (+139609us) tablet_replica.cc:205] Creating consensus instance
0805 22:23:40.922666 (+ 459us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":73}
I20260805 22:23:40.975445 31476 tablet_service.cc:1467] Processing CreateTablet for tablet cd72177fb2f24347ba19da011ccd3fe4 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260805 22:23:40.976663 31476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd72177fb2f24347ba19da011ccd3fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.970858 31697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d68a856424b4ceb8f4f54b4050e8dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.982038 31698 tablet_service.cc:1467] Processing CreateTablet for tablet eda619941ce64f788d0b7a364eed0bdb (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 982746729 <= VALUES < 1019144756
I20260805 22:23:40.970389 31779 consensus_queue.cc:260] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:40.989622 31696 tablet_service.cc:1467] Processing CreateTablet for tablet a260d4eadd0a4677aed088f331367c05 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1019144756 <= VALUES < 1055542783
I20260805 22:23:40.985395 31768 raft_consensus.cc:348] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:40.996594 31587 tablet_service.cc:1467] Processing CreateTablet for tablet f0eb14e6bb5c4855859507799f2b9e16 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260805 22:23:41.004770 31692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 18a7e330200a49349edcae379f1e2b57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.007069 31694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5df9cd03cde746e0956c8c7726d4eb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.013985 31696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a260d4eadd0a4677aed088f331367c05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.015226 31473 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1aa2dc5e5341f0a0450361d6c95814 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260805 22:23:41.016549 31473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1aa2dc5e5341f0a0450361d6c95814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.019299 31484 tablet_service.cc:1467] Processing CreateTablet for tablet 42324773ce144eb784e14693eecbee72 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260805 22:23:41.020802 31484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42324773ce144eb784e14693eecbee72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.999917 31768 raft_consensus.cc:374] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.027781 31698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eda619941ce64f788d0b7a364eed0bdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:40.985425 31472 tablet_service.cc:1467] Processing CreateTablet for tablet 1115c6a30f64413ab24c9d215d58a2c7 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260805 22:23:41.039220 31472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1115c6a30f64413ab24c9d215d58a2c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.042219 31693 tablet_service.cc:1467] Processing CreateTablet for tablet 4deaa72a885a43bfbc6171a7ed13ad81 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1055542783 <= VALUES < 1091940810
I20260805 22:23:41.043596 31693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4deaa72a885a43bfbc6171a7ed13ad81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.041026 31702 tablet_service.cc:1467] Processing CreateTablet for tablet 6f372cf6a3ec4a4cac2b6335a1a64a56 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1091940810 <= VALUES < 1128338837
I20260805 22:23:41.054315 31702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6f372cf6a3ec4a4cac2b6335a1a64a56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.033245 31768 raft_consensus.cc:729] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.031612 31476 tablet_service.cc:1467] Processing CreateTablet for tablet f914c8243e724eda849ce1fc2dc30827 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
I20260805 22:23:41.062703 31476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f914c8243e724eda849ce1fc2dc30827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.065186 31587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0eb14e6bb5c4855859507799f2b9e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.071159 31786 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:41.071295 31774 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:41.071736 31786 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.071813 31774 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.073971 31786 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:41.075515 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:41.070412 31485 tablet_service.cc:1467] Processing CreateTablet for tablet 8513af2d574d45b4a6d02428543b1402 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260805 22:23:41.077216 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:41.077862 31502 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
W20260805 22:23:41.079257 31554 leader_election.cc:341] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.079990 31481 tablet_service.cc:1467] Processing CreateTablet for tablet 16e20d6ca2f34a65a5d7bc4471282feb (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260805 22:23:41.080534 31554 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:41.074007 31774 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:41.082928 31774 raft_consensus.cc:2781] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:41.082901 31768 consensus_queue.cc:260] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.083530 31774 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:41.084000 31774 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:41.085147 31768 ts_tablet_manager.cc:1428] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.118s user 0.006s sys 0.000s
I20260805 22:23:41.090337 31695 tablet_service.cc:1467] Processing CreateTablet for tablet fe370453251f4300a8d90c397021595c (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1128338837 <= VALUES < 1164736864
I20260805 22:23:41.080942 31485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8513af2d574d45b4a6d02428543b1402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.096057 31484 tablet_service.cc:1467] Processing CreateTablet for tablet f8863c96c162438983f403d38b11c5db (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260805 22:23:41.102283 31484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8863c96c162438983f403d38b11c5db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.060231 31478 tablet_service.cc:1467] Processing CreateTablet for tablet 36ee93639c3f4084b94145c6ee60a2a8 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260805 22:23:41.097584 31768 tablet_bootstrap.cc:492] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:41.106997 31478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36ee93639c3f4084b94145c6ee60a2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.100464 31774 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.115119 31698 tablet_service.cc:1467] Processing CreateTablet for tablet 527d60ab6db74595bed7e87ca2ecc6e8 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1273930945 <= VALUES < 1310328972
I20260805 22:23:41.116420 31698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 527d60ab6db74595bed7e87ca2ecc6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.120137 31700 tablet_service.cc:1467] Processing CreateTablet for tablet 6b16a2e0814140e39609da87479bb57e (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1310328972 <= VALUES < 1346726999
I20260805 22:23:41.124382 31482 tablet_service.cc:1467] Processing CreateTablet for tablet 7c99e6a491454b5ba8d31e3246e77797 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260805 22:23:41.124783 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:41.125707 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:41.141147 31608 raft_consensus.cc:2379] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 1: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:23:41.126760 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:41.143278 31444 leader_election.cc:341] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.126646 31483 tablet_service.cc:1467] Processing CreateTablet for tablet fb8dc633587f48d1bba2a54a1619bc3e (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260805 22:23:41.127617 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:41.133391 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:41.134545 31771 tablet_bootstrap.cc:492] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:41.145300 31502 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:41.148250 31771 ts_tablet_manager.cc:1397] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.227s user 0.017s sys 0.052s
I20260805 22:23:41.145411 31484 tablet_service.cc:1467] Processing CreateTablet for tablet 17b3240dd1b947178dafdc9cac7f1e5e (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260805 22:23:41.151000 31771 raft_consensus.cc:348] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.151753 31771 raft_consensus.cc:374] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.152048 31771 raft_consensus.cc:729] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.152741 31771 consensus_queue.cc:260] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.136116 31774 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:41.137594 31481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16e20d6ca2f34a65a5d7bc4471282feb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.138329 31694 tablet_service.cc:1467] Processing CreateTablet for tablet 10de517cc0a24f20b5885091e99e252d (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1201134891 <= VALUES < 1237532918
I20260805 22:23:41.135555 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:41.154824 31703 tablet_service.cc:1467] Processing CreateTablet for tablet dcea14dac2934d139e2c5ab6d0b8648d (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1346726999 <= VALUES < 1383125026
I20260805 22:23:41.139132 31700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b16a2e0814140e39609da87479bb57e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260805 22:23:41.165222 31554 leader_election.cc:341] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.170076 31768 tablet_bootstrap.cc:654] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.175910 31795 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:41.176465 31795 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.178957 31795 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:41.182978 31702 tablet_service.cc:1467] Processing CreateTablet for tablet 986c28960d44411487f79ca134af59e2 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1383125026 <= VALUES < 1419523053
I20260805 22:23:41.179288 31478 tablet_service.cc:1467] Processing CreateTablet for tablet 9f41c2ff98f049c3bc654aa98cd77694 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260805 22:23:41.164237 31482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c99e6a491454b5ba8d31e3246e77797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.188657 31483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8dc633587f48d1bba2a54a1619bc3e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260805 22:23:41.189627 31444 leader_election.cc:341] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.190068 31444 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:41.190402 31502 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
W20260805 22:23:41.191284 31444 leader_election.cc:341] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.191680 31444 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:41.191701 31554 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:41.193442 31786 raft_consensus.cc:2726] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:41.193585 31774 raft_consensus.cc:2781] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:41.194068 31786 raft_consensus.cc:2726] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:41.194129 31774 raft_consensus.cc:686] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:41.195833 31774 consensus_queue.cc:237] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.196774 31472 tablet_service.cc:1467] Processing CreateTablet for tablet d8fd526bfd0047e49803dbf11f8eafc6 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260805 22:23:41.195906 31771 ts_tablet_manager.cc:1428] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.047s user 0.006s sys 0.000s
I20260805 22:23:41.200152 31695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe370453251f4300a8d90c397021595c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.211683 31484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17b3240dd1b947178dafdc9cac7f1e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.224095 31696 tablet_service.cc:1467] Processing CreateTablet for tablet c75e772a0f924080865ddf85ed1e2d86 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1419523053 <= VALUES < 1455921080
I20260805 22:23:41.230247 31779 ts_tablet_manager.cc:1428] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.370s user 0.014s sys 0.024s
I20260805 22:23:41.232357 31702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986c28960d44411487f79ca134af59e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.232357 31479 tablet_service.cc:1467] Processing CreateTablet for tablet 91a9acdbc98a4e799374fe6e8950c3a7 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260805 22:23:41.211205 31697 tablet_service.cc:1467] Processing CreateTablet for tablet 78cb483d17014b3281316012dbb6f6e1 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1237532918 <= VALUES < 1273930945
I20260805 22:23:41.240126 31473 tablet_service.cc:1467] Processing CreateTablet for tablet bd1c7254b2b74c10a4fb473ea633544d (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
W20260805 22:23:41.245759 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 54502.3us
I20260805 22:23:41.251789 31478 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f41c2ff98f049c3bc654aa98cd77694. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.251487 31358 catalog_manager.cc:5495] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:41.253803 31476 tablet_service.cc:1467] Processing CreateTablet for tablet e6cf90ae3fc840ceb48be8c790be97d9 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260805 22:23:41.254433 31768 tablet_bootstrap.cc:492] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:41.254906 31768 ts_tablet_manager.cc:1397] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.158s user 0.025s sys 0.008s
I20260805 22:23:41.255021 31476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6cf90ae3fc840ceb48be8c790be97d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.259068 31771 tablet_bootstrap.cc:492] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:41.259654 31472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8fd526bfd0047e49803dbf11f8eafc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.266047 31473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1c7254b2b74c10a4fb473ea633544d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.266359 31779 tablet_bootstrap.cc:492] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:41.270587 31479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a9acdbc98a4e799374fe6e8950c3a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.267424 31485 tablet_service.cc:1467] Processing CreateTablet for tablet f0eb14e6bb5c4855859507799f2b9e16 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260805 22:23:41.272773 31771 tablet_bootstrap.cc:654] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.273309 31768 raft_consensus.cc:348] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.274075 31768 raft_consensus.cc:374] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.274363 31768 raft_consensus.cc:729] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.275074 31768 consensus_queue.cc:260] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.277635 31693 tablet_service.cc:1467] Processing CreateTablet for tablet b185bcb30c384758983895f8e5d8c98a (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1492319107 <= VALUES < 1528717134
I20260805 22:23:41.275422 31692 tablet_service.cc:1467] Processing CreateTablet for tablet 1115c6a30f64413ab24c9d215d58a2c7 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1455921080 <= VALUES < 1492319107
I20260805 22:23:41.278040 31485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0eb14e6bb5c4855859507799f2b9e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.277285 31768 ts_tablet_manager.cc:1428] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20260805 22:23:41.297228 31768 tablet_bootstrap.cc:492] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:41.294799 31703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcea14dac2934d139e2c5ab6d0b8648d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.306671 31694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10de517cc0a24f20b5885091e99e252d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.307492 31693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b185bcb30c384758983895f8e5d8c98a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.310515 31700 tablet_service.cc:1467] Processing CreateTablet for tablet 6a1aa2dc5e5341f0a0450361d6c95814 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1565115161 <= VALUES < 1601513188
I20260805 22:23:41.312320 31700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1aa2dc5e5341f0a0450361d6c95814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.303844 31698 tablet_service.cc:1467] Processing CreateTablet for tablet cd72177fb2f24347ba19da011ccd3fe4 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1528717134 <= VALUES < 1565115161
I20260805 22:23:41.313367 31698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd72177fb2f24347ba19da011ccd3fe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.322595 31768 tablet_bootstrap.cc:654] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.325726 31702 tablet_service.cc:1467] Processing CreateTablet for tablet 42324773ce144eb784e14693eecbee72 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1601513188 <= VALUES < 1637911215
I20260805 22:23:41.338399 31771 tablet_bootstrap.cc:492] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:41.338950 31771 ts_tablet_manager.cc:1397] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.092s user 0.010s sys 0.002s
I20260805 22:23:41.342290 31771 raft_consensus.cc:348] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.343082 31771 raft_consensus.cc:374] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.343374 31771 raft_consensus.cc:729] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.344138 31771 consensus_queue.cc:260] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.338951 31702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42324773ce144eb784e14693eecbee72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.350574 31771 ts_tablet_manager.cc:1428] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.011s user 0.002s sys 0.004s
I20260805 22:23:41.352174 31774 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:41.352690 31774 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.354610 31774 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:41.357612 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:41.361037 31743 debug-util.cc:398] Leaking SignalData structure 0x7b08001f9640 after lost signal to thread 29786
W20260805 22:23:41.363924 31554 leader_election.cc:341] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:41.368201 31743 debug-util.cc:398] Leaking SignalData structure 0x7b08001ff200 after lost signal to thread 31402
I20260805 22:23:41.362744 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:41.366756 31692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1115c6a30f64413ab24c9d215d58a2c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.371871 31697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78cb483d17014b3281316012dbb6f6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.375689 31696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c75e772a0f924080865ddf85ed1e2d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.378504 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:23:41.380051 31667 leader_election.cc:341] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:41.381901 31554 leader_election.cc:341] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.382369 31554 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:41.383455 31774 raft_consensus.cc:2726] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:41.399296 31695 tablet_service.cc:1467] Processing CreateTablet for tablet 36ee93639c3f4084b94145c6ee60a2a8 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1674309242 <= VALUES < 1710707269
I20260805 22:23:41.400840 31695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36ee93639c3f4084b94145c6ee60a2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.402427 31779 tablet_bootstrap.cc:654] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.397187 31771 tablet_bootstrap.cc:492] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:41.414585 31693 tablet_service.cc:1467] Processing CreateTablet for tablet f914c8243e724eda849ce1fc2dc30827 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1637911215 <= VALUES < 1674309242
W20260805 22:23:41.415642 31423 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260805 22:23:41.415936 31693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f914c8243e724eda849ce1fc2dc30827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.429440 31771 tablet_bootstrap.cc:654] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.435092 31700 tablet_service.cc:1467] Processing CreateTablet for tablet 8513af2d574d45b4a6d02428543b1402 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1710707269 <= VALUES < 1747105296
I20260805 22:23:41.436465 31700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8513af2d574d45b4a6d02428543b1402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.439260 31697 tablet_service.cc:1467] Processing CreateTablet for tablet 16e20d6ca2f34a65a5d7bc4471282feb (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1747105296 <= VALUES < 1783503323
I20260805 22:23:41.440027 31779 tablet_bootstrap.cc:492] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:41.440552 31779 ts_tablet_manager.cc:1397] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.174s user 0.016s sys 0.000s
I20260805 22:23:41.440692 31697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16e20d6ca2f34a65a5d7bc4471282feb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.445317 31768 tablet_bootstrap.cc:492] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:41.445834 31768 ts_tablet_manager.cc:1397] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.149s user 0.040s sys 0.020s
I20260805 22:23:41.447115 31694 tablet_service.cc:1467] Processing CreateTablet for tablet fb8dc633587f48d1bba2a54a1619bc3e (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1783503323 <= VALUES < 1819901350
I20260805 22:23:41.447918 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:41.450263 31608 raft_consensus.cc:2454] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:41.451556 31667 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:23:41.452917 31799 raft_consensus.cc:2781] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:41.453302 31799 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:41.455068 31799 raft_consensus.cc:3037] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:41.448515 31768 raft_consensus.cc:348] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.455845 31768 raft_consensus.cc:374] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.456132 31768 raft_consensus.cc:729] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.457369 31779 raft_consensus.cc:348] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.458235 31779 raft_consensus.cc:374] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.458557 31779 raft_consensus.cc:729] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.457811 31768 consensus_queue.cc:260] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.459373 31694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8dc633587f48d1bba2a54a1619bc3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.463691 31692 tablet_service.cc:1467] Processing CreateTablet for tablet 7c99e6a491454b5ba8d31e3246e77797 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1856299377 <= VALUES < 1892697404
I20260805 22:23:41.465723 31801 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:41.465830 31799 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.466367 31801 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.467919 31799 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:41.468310 31801 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:41.469533 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:23:41.470963 31444 leader_election.cc:341] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.471132 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:41.471794 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:41.472366 31608 raft_consensus.cc:3037] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:41.475379 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:41.475711 31779 consensus_queue.cc:260] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.477866 31779 ts_tablet_manager.cc:1428] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.028s user 0.006s sys 0.000s
I20260805 22:23:41.479621 31768 ts_tablet_manager.cc:1428] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
I20260805 22:23:41.480576 31768 tablet_bootstrap.cc:492] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
W20260805 22:23:41.480927 31667 leader_election.cc:341] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:41.481978 31444 leader_election.cc:341] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.482399 31444 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:41.484045 31696 tablet_service.cc:1467] Processing CreateTablet for tablet f8863c96c162438983f403d38b11c5db (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1819901350 <= VALUES < 1856299377
I20260805 22:23:41.486370 31768 tablet_bootstrap.cc:654] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.488590 31608 raft_consensus.cc:2454] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:23:41.489899 31667 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:23:41.490813 31799 raft_consensus.cc:2781] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:41.491137 31801 raft_consensus.cc:2726] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:41.491290 31799 raft_consensus.cc:686] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:23:41.492257 31799 consensus_queue.cc:237] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.502192 31771 tablet_bootstrap.cc:492] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:41.502003 31695 tablet_service.cc:1467] Processing CreateTablet for tablet 17b3240dd1b947178dafdc9cac7f1e5e (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1892697404 <= VALUES < 1929095431
I20260805 22:23:41.503873 31771 ts_tablet_manager.cc:1397] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.107s user 0.015s sys 0.022s
I20260805 22:23:41.506258 31702 tablet_service.cc:1467] Processing CreateTablet for tablet d8fd526bfd0047e49803dbf11f8eafc6 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1929095431 <= VALUES < 1965493458
I20260805 22:23:41.506780 31771 raft_consensus.cc:348] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.512274 31771 raft_consensus.cc:374] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.513633 31771 raft_consensus.cc:729] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.514184 31697 tablet_service.cc:1467] Processing CreateTablet for tablet 91a9acdbc98a4e799374fe6e8950c3a7 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 2038289512 <= VALUES < 2074687539
I20260805 22:23:41.514488 31771 consensus_queue.cc:260] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.506865 31695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17b3240dd1b947178dafdc9cac7f1e5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.509024 31774 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:41.508797 31779 tablet_bootstrap.cc:492] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:41.516333 31696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f8863c96c162438983f403d38b11c5db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.516978 31702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8fd526bfd0047e49803dbf11f8eafc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.515938 31774 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.518669 31774 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:41.519512 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:41.519721 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:41.520150 31502 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
W20260805 22:23:41.521049 31554 leader_election.cc:341] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.524699 31779 tablet_bootstrap.cc:654] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.525318 31554 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:41.526530 31774 raft_consensus.cc:2781] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:41.526952 31774 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:41.527462 31774 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:41.528414 31698 tablet_service.cc:1467] Processing CreateTablet for tablet e6cf90ae3fc840ceb48be8c790be97d9 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 2074687539 <= VALUES < 2111085566
I20260805 22:23:41.529814 31698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6cf90ae3fc840ceb48be8c790be97d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.524654 31768 tablet_bootstrap.cc:492] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:41.533165 31768 ts_tablet_manager.cc:1397] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.053s user 0.025s sys 0.008s
I20260805 22:23:41.534572 31774 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.540191 31702 tablet_service.cc:1467] Processing CreateTablet for tablet f0eb14e6bb5c4855859507799f2b9e16 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 2111085566 <= VALUES
I20260805 22:23:41.527019 31771 ts_tablet_manager.cc:1428] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.023s user 0.005s sys 0.000s
I20260805 22:23:41.536053 31768 raft_consensus.cc:348] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.541052 31768 raft_consensus.cc:374] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.541363 31768 raft_consensus.cc:729] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.542013 31771 tablet_bootstrap.cc:492] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:41.542128 31768 consensus_queue.cc:260] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.543015 31703 tablet_service.cc:1467] Processing CreateTablet for tablet 9f41c2ff98f049c3bc654aa98cd77694 (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 1965493458 <= VALUES < 2001891485
I20260805 22:23:41.514077 31700 tablet_service.cc:1467] Processing CreateTablet for tablet bd1c7254b2b74c10a4fb473ea633544d (DEFAULT_TABLE table=test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]), partition=RANGE (key) PARTITION 2001891485 <= VALUES < 2038289512
I20260805 22:23:41.547116 31768 ts_tablet_manager.cc:1428] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.014s user 0.003s sys 0.003s
I20260805 22:23:41.548060 31768 tablet_bootstrap.cc:492] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:41.549666 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:41.550266 31502 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:41.550818 31779 tablet_bootstrap.cc:492] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:41.551322 31779 ts_tablet_manager.cc:1397] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.043s user 0.008s sys 0.001s
I20260805 22:23:41.550853 31692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c99e6a491454b5ba8d31e3246e77797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.553576 31697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a9acdbc98a4e799374fe6e8950c3a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.556280 31700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1c7254b2b74c10a4fb473ea633544d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.559005 31774 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:41.559628 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:41.561852 31771 tablet_bootstrap.cc:654] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.562532 31703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f41c2ff98f049c3bc654aa98cd77694. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.563684 31779 raft_consensus.cc:348] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.564525 31779 raft_consensus.cc:374] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.564792 31779 raft_consensus.cc:729] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.565676 31779 consensus_queue.cc:260] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.566520 31713 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:41.569051 31768 tablet_bootstrap.cc:654] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.569491 31702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0eb14e6bb5c4855859507799f2b9e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:41.573632 31713 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:41.573803 31357 catalog_manager.cc:5495] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:41.576918 31554 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:41.577903 31774 raft_consensus.cc:2781] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:41.578682 31774 raft_consensus.cc:686] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:41.579316 31502 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:41.579592 31774 consensus_queue.cc:237] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.591598 31798 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:41.595321 31779 ts_tablet_manager.cc:1428] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
I20260805 22:23:41.595925 31771 tablet_bootstrap.cc:492] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:41.596402 31771 ts_tablet_manager.cc:1397] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.055s user 0.009s sys 0.001s
I20260805 22:23:41.596328 31779 tablet_bootstrap.cc:492] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:41.601287 31771 raft_consensus.cc:348] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.602342 31771 raft_consensus.cc:374] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.605604 31771 raft_consensus.cc:729] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.606396 31771 consensus_queue.cc:260] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.610637 31771 ts_tablet_manager.cc:1428] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.014s user 0.008s sys 0.000s
I20260805 22:23:41.611680 31771 tablet_bootstrap.cc:492] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:41.619760 31771 tablet_bootstrap.cc:654] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.620088 31779 tablet_bootstrap.cc:654] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.615675 31357 catalog_manager.cc:5495] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:41.646282 31771 tablet_bootstrap.cc:492] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:41.646884 31771 ts_tablet_manager.cc:1397] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.035s user 0.010s sys 0.015s
I20260805 22:23:41.644227 31768 tablet_bootstrap.cc:492] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:41.649941 31768 ts_tablet_manager.cc:1397] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.102s user 0.021s sys 0.016s
I20260805 22:23:41.649811 31771 raft_consensus.cc:348] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.650605 31771 raft_consensus.cc:374] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.650899 31771 raft_consensus.cc:729] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.651621 31771 consensus_queue.cc:260] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.652922 31768 raft_consensus.cc:348] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.653832 31768 raft_consensus.cc:374] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.654148 31768 raft_consensus.cc:729] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.654361 31771 ts_tablet_manager.cc:1428] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260805 22:23:41.655406 31771 tablet_bootstrap.cc:492] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:41.654987 31768 consensus_queue.cc:260] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.657495 31768 ts_tablet_manager.cc:1428] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260805 22:23:41.658998 31768 tablet_bootstrap.cc:492] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:41.661669 31771 tablet_bootstrap.cc:654] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.662658 31779 tablet_bootstrap.cc:492] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:41.663123 31779 ts_tablet_manager.cc:1397] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.067s user 0.009s sys 0.002s
I20260805 22:23:41.666046 31768 tablet_bootstrap.cc:654] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.666184 31779 raft_consensus.cc:348] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.667104 31779 raft_consensus.cc:374] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.667431 31779 raft_consensus.cc:729] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.668300 31779 consensus_queue.cc:260] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.674929 31779 ts_tablet_manager.cc:1428] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260805 22:23:41.675985 31779 tablet_bootstrap.cc:492] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:41.676532 31801 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:23:41.689757 31554 consensus_peers.cc:487] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c -> Peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Couldn't send request to peer feb5e7ab4b1541beaf2674c3c5500385. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:41.689535 31801 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.691746 31801 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:41.692000 31771 tablet_bootstrap.cc:492] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:41.692559 31771 ts_tablet_manager.cc:1397] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.037s user 0.018s sys 0.011s
I20260805 22:23:41.694219 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:41.694953 31608 raft_consensus.cc:2454] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:41.695053 31771 raft_consensus.cc:348] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.695834 31771 raft_consensus.cc:374] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.696089 31771 raft_consensus.cc:729] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.696478 31444 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:41.696846 31771 consensus_queue.cc:260] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.697549 31786 raft_consensus.cc:2781] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:41.698122 31786 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:41.698473 31786 raft_consensus.cc:3037] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:41.701993 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:41.702729 31713 raft_consensus.cc:2454] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:41.715094 31786 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.723867 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:41.724592 31608 raft_consensus.cc:3037] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:41.729727 31801 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:41.730333 31801 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.733970 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:41.736858 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:41.738875 31444 leader_election.cc:341] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:41.739606 31444 leader_election.cc:341] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.739969 31444 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:41.740934 31801 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:41.741891 31801 raft_consensus.cc:2726] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:41.746533 31801 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:41.747143 31801 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.750641 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:23:41.752481 31444 leader_election.cc:341] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.755451 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:41.756173 31713 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:41.757937 31444 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:23:41.763298 31801 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:41.764858 31801 raft_consensus.cc:2781] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:41.765668 31801 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:41.766351 31801 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:41.774020 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:41.774931 31713 raft_consensus.cc:3037] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:41.783974 31786 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:41.783741 31608 raft_consensus.cc:2454] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:41.788938 31444 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:41.790206 31786 raft_consensus.cc:2781] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:41.790808 31786 raft_consensus.cc:686] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:41.791774 31786 consensus_queue.cc:237] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.796190 31779 tablet_bootstrap.cc:654] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.806110 31713 raft_consensus.cc:2454] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:41.812776 31768 tablet_bootstrap.cc:492] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:41.813418 31768 ts_tablet_manager.cc:1397] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.155s user 0.051s sys 0.097s
I20260805 22:23:41.816215 31768 raft_consensus.cc:348] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.817046 31768 raft_consensus.cc:374] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.817324 31768 raft_consensus.cc:729] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.818084 31768 consensus_queue.cc:260] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.825953 31768 ts_tablet_manager.cc:1428] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260805 22:23:41.827075 31768 tablet_bootstrap.cc:492] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:41.833863 31768 tablet_bootstrap.cc:654] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.835341 31847 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:41.835918 31847 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.838258 31847 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:41.840324 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:23:41.842626 31444 leader_election.cc:341] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.843658 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:41.845306 31444 leader_election.cc:341] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.845851 31444 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:41.851076 31847 raft_consensus.cc:2726] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:41.852949 31771 ts_tablet_manager.cc:1428] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.160s user 0.050s sys 0.108s
I20260805 22:23:41.854122 31771 tablet_bootstrap.cc:492] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:41.858321 31771 tablet_bootstrap.cc:654] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.858877 31845 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:41.864414 31768 tablet_bootstrap.cc:492] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:41.864910 31768 ts_tablet_manager.cc:1397] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.038s user 0.016s sys 0.000s
I20260805 22:23:41.864972 31801 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.867046 31801 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:41.867532 31768 raft_consensus.cc:348] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.868244 31768 raft_consensus.cc:374] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.859872 31845 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.868546 31768 raft_consensus.cc:729] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.868923 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:41.869550 31713 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:41.869235 31768 consensus_queue.cc:260] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.871348 31768 ts_tablet_manager.cc:1428] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260805 22:23:41.872550 31768 tablet_bootstrap.cc:492] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:41.873674 31779 tablet_bootstrap.cc:492] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:41.873703 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:41.874215 31779 ts_tablet_manager.cc:1397] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.198s user 0.022s sys 0.025s
I20260805 22:23:41.868441 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
W20260805 22:23:41.876315 31444 leader_election.cc:341] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.879232 31768 tablet_bootstrap.cc:654] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
W20260805 22:23:41.878149 31554 leader_election.cc:341] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.880337 31713 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:41.881503 31444 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:23:41.883775 31845 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:41.885166 31774 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:41.888916 31801 raft_consensus.cc:2781] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:41.889192 31774 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.889688 31801 raft_consensus.cc:686] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:41.890527 31801 consensus_queue.cc:237] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.891299 31774 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:41.886543 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:41.894016 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:41.894219 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:41.895829 31554 leader_election.cc:341] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.896546 31554 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:23:41.898054 31554 leader_election.cc:341] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:41.898451 31667 consensus_peers.cc:487] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 -> Peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Couldn't send request to peer f8e9f4c756da4203917c464e2b179994. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:41.898674 31845 raft_consensus.cc:2726] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:41.867443 31358 catalog_manager.cc:5495] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:41.899734 31779 raft_consensus.cc:348] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.900731 31779 raft_consensus.cc:374] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260805 22:23:41.899637 31554 leader_election.cc:341] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.901110 31779 raft_consensus.cc:729] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.902089 31554 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:41.902990 31845 raft_consensus.cc:2726] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:41.904525 31768 tablet_bootstrap.cc:492] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:41.905074 31768 ts_tablet_manager.cc:1397] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.033s user 0.015s sys 0.006s
I20260805 22:23:41.906755 31779 consensus_queue.cc:260] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.907879 31799 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:41.908352 31799 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.910185 31771 tablet_bootstrap.cc:492] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:41.910761 31771 ts_tablet_manager.cc:1397] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.057s user 0.008s sys 0.020s
I20260805 22:23:41.912310 31768 raft_consensus.cc:348] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.913103 31768 raft_consensus.cc:374] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.913370 31768 raft_consensus.cc:729] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.913539 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:41.915534 31768 consensus_queue.cc:260] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:41.916456 31667 leader_election.cc:341] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.916603 31799 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:41.917783 31768 ts_tablet_manager.cc:1428] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260805 22:23:41.918447 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:41.918773 31768 tablet_bootstrap.cc:492] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
W20260805 22:23:41.919996 31667 leader_election.cc:341] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:41.920426 31667 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:41.913508 31771 raft_consensus.cc:348] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.922427 31771 raft_consensus.cc:374] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.922750 31771 raft_consensus.cc:729] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.923520 31771 consensus_queue.cc:260] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.925999 31771 ts_tablet_manager.cc:1428] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260805 22:23:41.926937 31771 tablet_bootstrap.cc:492] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:41.928925 31768 tablet_bootstrap.cc:654] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.932739 31771 tablet_bootstrap.cc:654] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.923938 31799 raft_consensus.cc:2726] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:41.939373 31779 ts_tablet_manager.cc:1428] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.065s user 0.008s sys 0.000s
I20260805 22:23:41.940658 31779 tablet_bootstrap.cc:492] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:41.941128 31768 tablet_bootstrap.cc:492] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:41.941653 31768 ts_tablet_manager.cc:1397] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.007s
I20260805 22:23:41.944346 31768 raft_consensus.cc:348] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.945066 31768 raft_consensus.cc:374] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.945281 31771 tablet_bootstrap.cc:492] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:41.945871 31771 ts_tablet_manager.cc:1397] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.009s
I20260805 22:23:41.946519 31768 raft_consensus.cc:729] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.947296 31768 consensus_queue.cc:260] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.950901 31768 ts_tablet_manager.cc:1428] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.009s user 0.004s sys 0.002s
I20260805 22:23:41.951908 31768 tablet_bootstrap.cc:492] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:41.954572 31771 raft_consensus.cc:348] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.955245 31771 raft_consensus.cc:374] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:41.955490 31771 raft_consensus.cc:729] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:41.956051 31771 consensus_queue.cc:260] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:41.957885 31771 ts_tablet_manager.cc:1428] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.012s user 0.003s sys 0.002s
I20260805 22:23:41.958796 31771 tablet_bootstrap.cc:492] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:41.964375 31771 tablet_bootstrap.cc:654] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.967020 31768 tablet_bootstrap.cc:654] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.969687 31779 tablet_bootstrap.cc:654] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:41.979404 31359 catalog_manager.cc:5495] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:42.010172 31845 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.010771 31845 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.013307 31768 tablet_bootstrap.cc:492] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:42.013839 31768 ts_tablet_manager.cc:1397] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.062s user 0.022s sys 0.025s
I20260805 22:23:42.013739 31845 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.014571 31845 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.015049 31845 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.016213 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:42.016376 31768 raft_consensus.cc:348] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.017077 31768 raft_consensus.cc:374] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.017462 31768 raft_consensus.cc:729] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.018584 31771 tablet_bootstrap.cc:492] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:42.018183 31768 consensus_queue.cc:260] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.019074 31771 ts_tablet_manager.cc:1397] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.050s
I20260805 22:23:42.020354 31768 ts_tablet_manager.cc:1428] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260805 22:23:42.021530 31768 tablet_bootstrap.cc:492] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:42.017048 31845 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.021541 31771 raft_consensus.cc:348] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.022653 31771 raft_consensus.cc:374] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.022888 31771 raft_consensus.cc:729] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.023085 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:42.023488 31771 consensus_queue.cc:260] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.024855 31801 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.025554 31771 ts_tablet_manager.cc:1428] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260805 22:23:42.025329 31801 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.026418 31771 tablet_bootstrap.cc:492] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:42.027556 31801 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.030315 31845 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.030772 31845 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.032816 31845 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.033797 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:42.034545 31502 raft_consensus.cc:2454] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:42.035053 31799 consensus_queue.cc:1035] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:42.036058 31768 tablet_bootstrap.cc:654] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
W20260805 22:23:42.021955 31554 leader_election.cc:341] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.036842 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:42.037585 31608 raft_consensus.cc:2454] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
W20260805 22:23:42.038074 31554 leader_election.cc:341] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.038940 31444 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:42.042315 31786 raft_consensus.cc:2781] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:42.042687 31786 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:42.043015 31786 raft_consensus.cc:3037] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.039264 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:42.040068 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:42.018447 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:42.047295 31444 leader_election.cc:341] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.053877 31554 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:23:42.055348 31554 leader_election.cc:341] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:42.056149 31554 leader_election.cc:341] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.057190 31554 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:42.056681 31845 raft_consensus.cc:2781] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:42.059355 31845 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:42.059744 31845 raft_consensus.cc:3037] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.041069 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:42.065980 31502 raft_consensus.cc:2454] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:42.067713 31774 raft_consensus.cc:2726] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:42.069144 31554 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:42.077311 31774 raft_consensus.cc:2781] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:42.077749 31774 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:42.078122 31774 raft_consensus.cc:3037] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.087050 31798 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.086937 31774 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.087653 31798 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.090724 31798 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.090849 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:42.074626 31771 tablet_bootstrap.cc:654] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.091555 31502 raft_consensus.cc:3037] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.052762 31786 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.097704 31786 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.098420 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:23:42.099967 31444 leader_election.cc:341] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.101131 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:42.097208 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:42.104200 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:42.104921 31501 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:42.108278 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:42.108552 31771 tablet_bootstrap.cc:492] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:42.108996 31771 ts_tablet_manager.cc:1397] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.083s user 0.007s sys 0.003s
I20260805 22:23:42.111677 31771 raft_consensus.cc:348] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.112413 31771 raft_consensus.cc:374] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.112679 31771 raft_consensus.cc:729] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.113397 31771 consensus_queue.cc:260] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.115353 31771 ts_tablet_manager.cc:1428] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
W20260805 22:23:42.115692 31554 leader_election.cc:341] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.116216 31771 tablet_bootstrap.cc:492] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:42.120906 31768 tablet_bootstrap.cc:492] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:42.121435 31768 ts_tablet_manager.cc:1397] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.100s user 0.032s sys 0.042s
I20260805 22:23:42.124022 31768 raft_consensus.cc:348] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.124769 31768 raft_consensus.cc:374] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.125029 31768 raft_consensus.cc:729] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.125707 31801 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.126185 31801 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.128225 31801 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.130191 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:42.131807 31502 raft_consensus.cc:2454] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:42.132961 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:42.133798 31845 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.134678 31444 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:42.135192 31608 raft_consensus.cc:2379] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 1: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:23:42.135939 31801 raft_consensus.cc:2726] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:42.089282 31774 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.136715 31444 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:42.136960 31768 consensus_queue.cc:260] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.137956 31801 raft_consensus.cc:2726] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260805 22:23:42.138712 31768 ts_tablet_manager.cc:1428] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260805 22:23:42.139361 31768 tablet_bootstrap.cc:492] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:42.136073 31845 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:23:42.120472 31554 leader_election.cc:341] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.145541 31768 tablet_bootstrap.cc:654] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.146497 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:42.145557 31779 tablet_bootstrap.cc:492] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:42.155579 31779 ts_tablet_manager.cc:1397] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.215s user 0.028s sys 0.102s
I20260805 22:23:42.156638 31768 tablet_bootstrap.cc:492] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:42.157003 31768 ts_tablet_manager.cc:1397] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.001s
I20260805 22:23:42.147524 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:42.158157 31502 raft_consensus.cc:2379] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 1: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:23:42.158347 31779 raft_consensus.cc:348] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.158805 31768 raft_consensus.cc:348] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.159065 31779 raft_consensus.cc:374] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.146152 31554 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:42.159363 31768 raft_consensus.cc:374] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.159401 31779 raft_consensus.cc:729] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.159693 31768 raft_consensus.cc:729] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.146771 31771 tablet_bootstrap.cc:654] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.160272 31768 consensus_queue.cc:260] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.160290 31779 consensus_queue.cc:260] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.162123 31768 ts_tablet_manager.cc:1428] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
I20260805 22:23:42.159858 31774 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:42.162721 31774 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:42.163066 31774 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.163311 31768 tablet_bootstrap.cc:492] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:42.169446 31768 tablet_bootstrap.cc:654] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.170020 31554 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:42.170418 31779 ts_tablet_manager.cc:1428] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20260805 22:23:42.170869 31845 raft_consensus.cc:2781] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:42.171370 31845 raft_consensus.cc:686] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:42.172593 31845 consensus_queue.cc:237] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.173305 31798 consensus_queue.cc:1035] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:42.175207 31779 tablet_bootstrap.cc:492] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:42.175514 31771 tablet_bootstrap.cc:492] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:42.175894 31771 ts_tablet_manager.cc:1397] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.060s user 0.012s sys 0.000s
I20260805 22:23:42.173398 31774 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.178849 31771 raft_consensus.cc:348] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.184768 31771 raft_consensus.cc:374] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.185089 31771 raft_consensus.cc:729] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.185817 31771 consensus_queue.cc:260] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.187704 31774 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.194365 31768 tablet_bootstrap.cc:492] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:42.194851 31768 ts_tablet_manager.cc:1397] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.011s
I20260805 22:23:42.197283 31768 raft_consensus.cc:348] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.197962 31768 raft_consensus.cc:374] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.188163 31771 ts_tablet_manager.cc:1428] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
W20260805 22:23:42.192152 31554 leader_election.cc:341] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.198164 31779 tablet_bootstrap.cc:654] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.198233 31768 raft_consensus.cc:729] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.198527 31554 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:42.198935 31768 consensus_queue.cc:260] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.199936 31774 raft_consensus.cc:2726] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260805 22:23:42.201032 31768 ts_tablet_manager.cc:1428] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260805 22:23:42.200717 31359 catalog_manager.cc:5495] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:42.201982 31768 tablet_bootstrap.cc:492] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:42.203208 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:42.203799 31502 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.207477 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:23:42.209543 31554 leader_election.cc:341] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.210179 31774 consensus_queue.cc:1035] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:42.221629 31771 tablet_bootstrap.cc:492] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:42.227480 31768 tablet_bootstrap.cc:654] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.227821 31779 tablet_bootstrap.cc:492] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:42.228425 31779 ts_tablet_manager.cc:1397] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.053s user 0.024s sys 0.010s
I20260805 22:23:42.232885 31779 raft_consensus.cc:348] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.233832 31779 raft_consensus.cc:374] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.234189 31779 raft_consensus.cc:729] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.236070 31801 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.236505 31768 tablet_bootstrap.cc:492] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:42.237063 31768 ts_tablet_manager.cc:1397] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.035s user 0.007s sys 0.005s
I20260805 22:23:42.236555 31801 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.238730 31801 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.239918 31502 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:42.241309 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:42.242003 31608 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:42.243260 31768 raft_consensus.cc:348] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.243621 31444 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:42.243947 31768 raft_consensus.cc:374] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.244257 31768 raft_consensus.cc:729] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.244556 31801 raft_consensus.cc:2781] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:42.245118 31801 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:42.244880 31768 consensus_queue.cc:260] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.245767 31801 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.245818 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:42.246490 31714 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:42.246979 31768 ts_tablet_manager.cc:1428] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20260805 22:23:42.247882 31768 tablet_bootstrap.cc:492] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:42.250147 31774 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.250631 31774 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.252444 31774 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.252755 31786 consensus_queue.cc:1035] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:42.253165 31801 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.254608 31771 tablet_bootstrap.cc:654] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.255059 31801 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.256003 31768 tablet_bootstrap.cc:654] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.255925 31779 consensus_queue.cc:260] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.258582 31779 ts_tablet_manager.cc:1428] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.030s user 0.004s sys 0.003s
I20260805 22:23:42.259523 31779 tablet_bootstrap.cc:492] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:42.265202 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:42.265862 31608 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.270025 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:42.270648 31713 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.270732 31771 tablet_bootstrap.cc:492] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:42.271292 31771 ts_tablet_manager.cc:1397] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.050s user 0.009s sys 0.003s
I20260805 22:23:42.274639 31771 raft_consensus.cc:348] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.275383 31771 raft_consensus.cc:374] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.275652 31771 raft_consensus.cc:729] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.278452 31779 tablet_bootstrap.cc:654] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.277366 31554 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:42.280341 31845 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:42.280792 31845 raft_consensus.cc:686] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:42.281674 31845 consensus_queue.cc:237] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.288460 31771 consensus_queue.cc:260] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.290928 31771 ts_tablet_manager.cc:1428] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.019s user 0.006s sys 0.002s
I20260805 22:23:42.295229 31713 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:42.297205 31798 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.300081 31798 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.297883 31774 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.299445 31865 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.303149 31774 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.307989 31774 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.308377 31768 tablet_bootstrap.cc:492] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:42.303248 31865 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.308964 31768 ts_tablet_manager.cc:1397] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.061s user 0.019s sys 0.031s
I20260805 22:23:42.310156 31771 tablet_bootstrap.cc:492] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:42.312103 31774 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.312979 31444 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:42.313253 31774 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.314146 31865 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:42.314813 31798 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.315284 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:42.315541 31774 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.315970 31609 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:42.316871 31801 raft_consensus.cc:2781] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:42.317610 31667 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:42.318571 31865 raft_consensus.cc:2781] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:42.318922 31865 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:42.318858 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:42.319373 31865 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.320236 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:42.320863 31502 raft_consensus.cc:2454] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:42.321333 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:42.321787 31501 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:42.304299 31608 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:42.323956 31768 raft_consensus.cc:348] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.326122 31768 raft_consensus.cc:374] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.326418 31768 raft_consensus.cc:729] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.327394 31768 consensus_queue.cc:260] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.320858 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:42.328970 31499 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:42.329303 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:42.329759 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:42.332909 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:42.333359 31786 consensus_queue.cc:1035] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:42.335229 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:42.335937 31713 raft_consensus.cc:2454] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:42.338932 31359 catalog_manager.cc:5495] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:42.322862 31801 raft_consensus.cc:686] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:42.342358 31768 ts_tablet_manager.cc:1428] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.033s user 0.006s sys 0.001s
I20260805 22:23:42.362968 31768 tablet_bootstrap.cc:492] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:42.359292 31845 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.331163 31554 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
W20260805 22:23:42.401086 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 73913.1us
I20260805 22:23:42.400966 31801 consensus_queue.cc:237] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.402275 31774 raft_consensus.cc:2781] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:42.402750 31774 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:42.403090 31774 raft_consensus.cc:3037] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.407097 31779 tablet_bootstrap.cc:492] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:42.410240 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:23:42.407166 31554 leader_election.cc:341] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:42.411367 31554 leader_election.cc:341] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:42.412650 31554 leader_election.cc:341] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.413043 31554 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:42.413211 31786 consensus_queue.cc:1035] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:42.413874 31798 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260805 22:23:42.419862 31444 consensus_peers.cc:487] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 -> Peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Couldn't send request to peer 516cb07ce3064ccbbca35e912d62816c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260805 22:23:42.425498 31423 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260805 22:23:42.429303 31554 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:42.430295 31798 raft_consensus.cc:2781] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:42.430692 31798 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:42.431025 31798 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.407531 31845 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.435173 31845 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.438048 31779 ts_tablet_manager.cc:1397] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.178s user 0.020s sys 0.026s
W20260805 22:23:42.448256 31554 leader_election.cc:341] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.448874 31554 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:42.450351 31865 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.451670 31711 raft_consensus.cc:2379] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 1: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:23:42.454483 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:42.455101 31500 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.456194 31779 raft_consensus.cc:348] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.457001 31779 raft_consensus.cc:374] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.457316 31779 raft_consensus.cc:729] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.455217 31771 tablet_bootstrap.cc:654] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.457573 31845 raft_consensus.cc:2726] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:42.458554 31779 consensus_queue.cc:260] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.460657 31779 ts_tablet_manager.cc:1428] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.022s user 0.004s sys 0.000s
I20260805 22:23:42.463949 31779 tablet_bootstrap.cc:492] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:42.469668 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:42.470331 31768 tablet_bootstrap.cc:654] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.470491 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:42.474402 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:42.479755 31779 tablet_bootstrap.cc:654] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.471566 31865 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:42.475416 31774 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.485653 31774 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.485532 31358 catalog_manager.cc:5495] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
W20260805 22:23:42.472818 31554 leader_election.cc:341] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.488683 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
W20260805 22:23:42.488896 31554 leader_election.cc:341] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.490130 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:42.490988 31554 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:42.480049 31798 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.489306 31499 raft_consensus.cc:3037] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.491258 31711 raft_consensus.cc:3037] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.493618 31609 raft_consensus.cc:2379] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 1: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:23:42.494343 31774 raft_consensus.cc:2726] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:42.500903 31499 raft_consensus.cc:2454] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:42.505041 31798 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.506141 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:42.503829 31711 raft_consensus.cc:2454] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:42.506783 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:42.520790 31714 raft_consensus.cc:2379] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 1: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:23:42.507596 31554 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:42.528118 31501 raft_consensus.cc:3037] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.533932 31798 raft_consensus.cc:2781] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:42.534435 31798 raft_consensus.cc:686] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:42.535166 31554 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:42.535291 31798 consensus_queue.cc:237] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.530686 31500 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:23:42.538760 31895 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.538775 31801 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.539227 31667 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:23:42.540149 31865 raft_consensus.cc:2781] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:42.539834 31801 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.539271 31895 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.538530 31774 raft_consensus.cc:2726] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260805 22:23:42.540768 31865 raft_consensus.cc:686] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:23:42.541702 31865 consensus_queue.cc:237] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.542745 31895 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.545030 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:42.549234 31608 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:42.549340 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:42.549583 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:42.549979 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:42.551462 31609 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:42.553320 31444 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:42.554212 31895 raft_consensus.cc:2781] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:42.554576 31895 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:42.554932 31895 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.542795 31801 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.562278 31444 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
W20260805 22:23:42.563354 31444 leader_election.cc:341] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:42.564397 31444 leader_election.cc:341] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.565275 31801 raft_consensus.cc:2781] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:42.565706 31801 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:42.566031 31801 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.594059 31768 tablet_bootstrap.cc:492] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:42.594511 31768 ts_tablet_manager.cc:1397] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.232s user 0.024s sys 0.033s
I20260805 22:23:42.596716 31779 tablet_bootstrap.cc:492] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:42.597314 31779 ts_tablet_manager.cc:1397] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.134s user 0.014s sys 0.003s
I20260805 22:23:42.590066 31895 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.600015 31779 raft_consensus.cc:348] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.600754 31779 raft_consensus.cc:374] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.601014 31779 raft_consensus.cc:729] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.601936 31771 tablet_bootstrap.cc:492] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:42.602397 31771 ts_tablet_manager.cc:1397] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.292s user 0.015s sys 0.026s
I20260805 22:23:42.602253 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:42.602906 31608 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.605019 31771 raft_consensus.cc:348] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.605990 31771 raft_consensus.cc:374] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.606302 31771 raft_consensus.cc:729] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.607039 31771 consensus_queue.cc:260] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.606398 31801 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.607760 31357 catalog_manager.cc:5495] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:42.609086 31771 ts_tablet_manager.cc:1428] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260805 22:23:42.609762 31801 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.610072 31771 tablet_bootstrap.cc:492] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:42.611047 31608 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:42.596644 31768 raft_consensus.cc:348] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.612097 31768 raft_consensus.cc:374] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.612353 31768 raft_consensus.cc:729] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.612982 31768 consensus_queue.cc:260] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.613263 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:42.615051 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:42.615617 31609 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.624495 31444 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:42.624848 31771 tablet_bootstrap.cc:654] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.625599 31768 ts_tablet_manager.cc:1428] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.031s user 0.007s sys 0.002s
I20260805 22:23:42.624668 31779 consensus_queue.cc:260] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:42.625638 31444 leader_election.cc:341] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.644641 31779 ts_tablet_manager.cc:1428] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.047s user 0.008s sys 0.000s
I20260805 22:23:42.644665 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:42.645622 31779 tablet_bootstrap.cc:492] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
W20260805 22:23:42.646386 31444 leader_election.cc:341] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.628458 31768 tablet_bootstrap.cc:492] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:42.632794 31895 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.633244 31801 raft_consensus.cc:2781] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:42.635130 31774 consensus_queue.cc:1035] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:42.647789 31801 raft_consensus.cc:686] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:42.628153 31609 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:42.642721 31771 tablet_bootstrap.cc:492] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:42.648830 31801 consensus_queue.cc:237] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.650591 31771 ts_tablet_manager.cc:1397] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.041s user 0.011s sys 0.001s
W20260805 22:23:42.651947 31554 consensus_peers.cc:487] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c -> Peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Couldn't send request to peer feb5e7ab4b1541beaf2674c3c5500385. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:42.652678 31444 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:42.653522 31895 raft_consensus.cc:2781] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:42.654256 31895 raft_consensus.cc:686] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:42.653683 31357 catalog_manager.cc:5495] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:42.655033 31895 consensus_queue.cc:237] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.665733 31768 tablet_bootstrap.cc:654] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.667524 31771 raft_consensus.cc:348] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.668331 31771 raft_consensus.cc:374] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.668593 31771 raft_consensus.cc:729] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.672159 31771 consensus_queue.cc:260] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.680800 31771 ts_tablet_manager.cc:1428] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.030s user 0.005s sys 0.002s
I20260805 22:23:42.681826 31771 tablet_bootstrap.cc:492] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:42.688447 31768 tablet_bootstrap.cc:492] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:42.689662 31768 ts_tablet_manager.cc:1397] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.061s user 0.004s sys 0.014s
I20260805 22:23:42.690757 31774 consensus_queue.cc:1035] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260805 22:23:42.694008 31554 consensus_peers.cc:487] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c -> Peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Couldn't send request to peer feb5e7ab4b1541beaf2674c3c5500385. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:42.695977 31779 tablet_bootstrap.cc:654] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.696099 31768 raft_consensus.cc:348] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.696897 31768 raft_consensus.cc:374] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.697163 31768 raft_consensus.cc:729] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.697988 31768 consensus_queue.cc:260] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.699661 31771 tablet_bootstrap.cc:654] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.699806 31358 catalog_manager.cc:5495] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:42.701514 31358 catalog_manager.cc:5495] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:42.711707 31711 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.723920 31768 ts_tablet_manager.cc:1428] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.034s user 0.000s sys 0.005s
I20260805 22:23:42.731555 31768 tablet_bootstrap.cc:492] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:42.750715 31768 tablet_bootstrap.cc:654] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.756244 31771 tablet_bootstrap.cc:492] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:42.756764 31771 ts_tablet_manager.cc:1397] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.075s user 0.031s sys 0.016s
I20260805 22:23:42.759361 31771 raft_consensus.cc:348] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.760116 31771 raft_consensus.cc:374] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.760376 31771 raft_consensus.cc:729] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.760856 31779 tablet_bootstrap.cc:492] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:42.761404 31779 ts_tablet_manager.cc:1397] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.116s user 0.008s sys 0.009s
I20260805 22:23:42.761102 31771 consensus_queue.cc:260] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.762867 31895 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.763358 31895 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.765483 31895 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.765215 31779 raft_consensus.cc:348] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.766629 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:42.767285 31609 raft_consensus.cc:2454] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:42.768553 31444 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:42.768975 31798 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.769694 31895 raft_consensus.cc:2781] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:42.769414 31798 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.771591 31895 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:42.771751 31798 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.771950 31771 ts_tablet_manager.cc:1428] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260805 22:23:42.772089 31895 raft_consensus.cc:3037] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.773057 31771 tablet_bootstrap.cc:492] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:42.773708 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:42.774456 31714 raft_consensus.cc:2454] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:42.774487 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:42.766629 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:42.775874 31500 raft_consensus.cc:2454] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:42.776808 31554 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:42.777737 31798 raft_consensus.cc:2781] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:42.778065 31798 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:42.778374 31798 raft_consensus.cc:3037] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.779675 31771 tablet_bootstrap.cc:654] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
W20260805 22:23:42.776883 31444 leader_election.cc:341] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.766129 31779 raft_consensus.cc:374] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.792239 31779 raft_consensus.cc:729] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.793902 31845 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.794379 31845 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.792654 31798 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.796301 31845 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.796386 31798 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.798693 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:42.799033 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:42.799571 31714 raft_consensus.cc:3037] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.802668 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:42.803277 31501 raft_consensus.cc:3037] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.793357 31779 consensus_queue.cc:260] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.802695 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:23:42.806619 31554 leader_election.cc:341] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.807796 31711 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:42.811105 31554 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:23:42.812315 31845 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.812381 31714 raft_consensus.cc:2454] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:42.812795 31845 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.813728 31554 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:42.814527 31774 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:42.814615 31798 raft_consensus.cc:2781] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:42.814934 31774 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:42.815321 31774 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.817857 31501 raft_consensus.cc:2454] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:42.819499 31798 raft_consensus.cc:686] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:42.820430 31798 consensus_queue.cc:237] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.824187 31774 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.827206 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:42.825012 31845 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.826452 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:23:42.828500 31554 leader_election.cc:341] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:42.829268 31554 leader_election.cc:341] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.829738 31554 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:42.830791 31845 raft_consensus.cc:2726] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:42.831758 31771 tablet_bootstrap.cc:492] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:42.832347 31771 ts_tablet_manager.cc:1397] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.059s user 0.017s sys 0.040s
I20260805 22:23:42.832597 31774 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.833897 31895 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.836777 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:42.837373 31609 raft_consensus.cc:3037] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.839360 31771 raft_consensus.cc:348] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.840158 31771 raft_consensus.cc:374] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.840462 31771 raft_consensus.cc:729] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.841512 31771 consensus_queue.cc:260] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.843885 31771 ts_tablet_manager.cc:1428] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260805 22:23:42.847811 31771 tablet_bootstrap.cc:492] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:42.848259 31801 consensus_queue.cc:1035] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:42.853462 31771 tablet_bootstrap.cc:654] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.854383 31609 raft_consensus.cc:2454] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:42.855489 31444 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:42.856395 31801 raft_consensus.cc:2781] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:42.857012 31801 raft_consensus.cc:686] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:42.857896 31801 consensus_queue.cc:237] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.871419 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:42.872113 31501 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.876374 31768 tablet_bootstrap.cc:492] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:42.876911 31768 ts_tablet_manager.cc:1397] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.146s user 0.061s sys 0.053s
I20260805 22:23:42.877020 31779 ts_tablet_manager.cc:1428] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.115s user 0.005s sys 0.002s
I20260805 22:23:42.877766 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:42.878392 31714 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.879591 31768 raft_consensus.cc:348] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.880334 31768 raft_consensus.cc:374] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.880594 31768 raft_consensus.cc:729] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.881322 31768 consensus_queue.cc:260] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.882985 31895 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:42.883450 31768 ts_tablet_manager.cc:1428] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20260805 22:23:42.884472 31768 tablet_bootstrap.cc:492] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:42.884388 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:23:42.885931 31444 leader_election.cc:341] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:42.880678 31779 tablet_bootstrap.cc:492] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:42.889259 31714 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:42.892800 31501 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:42.894133 31554 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:42.894680 31768 tablet_bootstrap.cc:654] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.895515 31798 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:42.897281 31798 raft_consensus.cc:686] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:42.898197 31798 consensus_queue.cc:237] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.905603 31779 tablet_bootstrap.cc:654] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.910604 31771 tablet_bootstrap.cc:492] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:42.911015 31771 ts_tablet_manager.cc:1397] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.063s user 0.024s sys 0.030s
I20260805 22:23:42.942350 31771 raft_consensus.cc:348] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.916864 31358 catalog_manager.cc:5495] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:42.943094 31771 raft_consensus.cc:374] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:42.943365 31771 raft_consensus.cc:729] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:42.943907 31771 consensus_queue.cc:260] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:42.915812 31359 catalog_manager.cc:5495] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:42.912533 31847 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:42.914125 31895 consensus_queue.cc:1035] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.031s
I20260805 22:23:42.946403 31771 ts_tablet_manager.cc:1428] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.035s user 0.017s sys 0.008s
I20260805 22:23:42.947439 31771 tablet_bootstrap.cc:492] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:42.949826 31798 consensus_queue.cc:1035] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:42.956951 31779 tablet_bootstrap.cc:492] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:42.957592 31779 ts_tablet_manager.cc:1397] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.077s user 0.010s sys 0.008s
I20260805 22:23:42.964725 31608 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:42.997218 31771 tablet_bootstrap.cc:654] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:42.954552 31847 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.005074 31779 raft_consensus.cc:348] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.008673 31779 raft_consensus.cc:374] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.008965 31779 raft_consensus.cc:729] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.010262 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:43.011176 31609 raft_consensus.cc:2454] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:43.011768 31798 consensus_queue.cc:1035] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:43.013587 31444 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:43.014489 31801 raft_consensus.cc:2781] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:43.020397 31779 consensus_queue.cc:260] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.014818 31801 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:43.025890 31768 tablet_bootstrap.cc:492] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:43.026409 31768 ts_tablet_manager.cc:1397] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.142s user 0.044s sys 0.019s
I20260805 22:23:43.033371 31768 raft_consensus.cc:348] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.034188 31768 raft_consensus.cc:374] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.034438 31768 raft_consensus.cc:729] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.035669 31799 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:43.036158 31799 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.036660 31779 ts_tablet_manager.cc:1428] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.079s user 0.008s sys 0.000s
I20260805 22:23:43.027540 31801 raft_consensus.cc:3037] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.041538 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:43.044283 31444 leader_election.cc:341] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:43.045672 31779 tablet_bootstrap.cc:492] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:43.047519 31799 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:43.047921 31895 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:43.047470 31768 consensus_queue.cc:260] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.049310 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:43.052572 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:43.054232 31895 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.054715 31609 raft_consensus.cc:2379] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 1: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:23:43.056133 31667 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:43.056135 31895 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.058599 31768 ts_tablet_manager.cc:1428] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20260805 22:23:43.059945 31768 tablet_bootstrap.cc:492] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:43.062856 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:43.063511 31609 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:43.029558 31847 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.065196 31799 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.069878 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:43.079006 31444 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
W20260805 22:23:43.080049 31444 leader_election.cc:341] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:43.082327 31768 tablet_bootstrap.cc:654] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.085655 31895 consensus_queue.cc:1035] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:43.099635 31801 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.105778 31801 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.107100 31779 tablet_bootstrap.cc:654] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.110397 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:43.090762 31847 raft_consensus.cc:2781] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:43.112530 31604 raft_consensus.cc:3037] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.118136 31801 consensus_queue.cc:1035] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:43.115296 31799 raft_consensus.cc:2726] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:43.119372 31847 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:43.139947 31847 raft_consensus.cc:3037] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.151379 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:43.160351 31604 raft_consensus.cc:2454] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:43.160897 31771 tablet_bootstrap.cc:492] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:43.165465 31771 ts_tablet_manager.cc:1397] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.218s user 0.017s sys 0.015s
I20260805 22:23:43.161345 31359 catalog_manager.cc:5495] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
W20260805 22:23:43.162659 31444 leader_election.cc:341] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:43.161579 31847 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.165946 31799 consensus_queue.cc:1035] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:43.174661 31865 consensus_queue.cc:1035] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:43.174976 31444 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:43.176909 31801 raft_consensus.cc:2781] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:43.182840 31801 raft_consensus.cc:686] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:43.183653 31801 consensus_queue.cc:237] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.185003 31779 tablet_bootstrap.cc:492] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:43.185513 31779 ts_tablet_manager.cc:1397] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.140s user 0.012s sys 0.006s
I20260805 22:23:43.188005 31779 raft_consensus.cc:348] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.188704 31779 raft_consensus.cc:374] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.188952 31779 raft_consensus.cc:729] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.193583 31771 raft_consensus.cc:348] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.194278 31771 raft_consensus.cc:374] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.194535 31771 raft_consensus.cc:729] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.195133 31771 consensus_queue.cc:260] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:43.200099 31444 consensus_peers.cc:487] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 -> Peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Couldn't send request to peer feb5e7ab4b1541beaf2674c3c5500385. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:43.201537 31771 ts_tablet_manager.cc:1428] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s
I20260805 22:23:43.201519 31847 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.209210 31771 tablet_bootstrap.cc:492] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:43.214136 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:43.214524 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:43.214805 31604 raft_consensus.cc:3037] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.202013 31779 consensus_queue.cc:260] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.222208 31708 raft_consensus.cc:3037] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.228108 31768 tablet_bootstrap.cc:492] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:43.228590 31768 ts_tablet_manager.cc:1397] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.169s user 0.016s sys 0.007s
I20260805 22:23:43.228065 31604 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:43.229252 31708 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:43.230481 31444 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:43.231257 31786 raft_consensus.cc:2781] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:43.231758 31786 raft_consensus.cc:686] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:43.232601 31786 consensus_queue.cc:237] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.241889 31768 raft_consensus.cc:348] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.242640 31768 raft_consensus.cc:374] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.242893 31768 raft_consensus.cc:729] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
W20260805 22:23:43.243700 31444 consensus_peers.cc:487] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 -> Peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Couldn't send request to peer feb5e7ab4b1541beaf2674c3c5500385. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:43.243505 31768 consensus_queue.cc:260] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.245661 31768 ts_tablet_manager.cc:1428] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20260805 22:23:43.246488 31768 tablet_bootstrap.cc:492] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:43.247422 31847 consensus_queue.cc:1035] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:43.252230 31768 tablet_bootstrap.cc:654] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.262683 31779 ts_tablet_manager.cc:1428] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.077s user 0.009s sys 0.000s
I20260805 22:23:43.263499 31779 tablet_bootstrap.cc:492] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:43.272356 31779 tablet_bootstrap.cc:654] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.278504 31359 catalog_manager.cc:5495] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:43.291236 31845 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:43.293759 31845 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.298619 31845 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.300969 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:23:43.302464 31554 leader_election.cc:341] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:43.304993 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:43.306779 31554 leader_election.cc:341] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:43.307420 31554 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:43.309592 31358 catalog_manager.cc:5495] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:43.310933 31845 raft_consensus.cc:2726] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:43.314380 31779 tablet_bootstrap.cc:492] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:43.314919 31779 ts_tablet_manager.cc:1397] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.005s
I20260805 22:23:43.314930 31771 tablet_bootstrap.cc:654] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.315577 31768 tablet_bootstrap.cc:492] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:43.316128 31768 ts_tablet_manager.cc:1397] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.070s user 0.011s sys 0.016s
I20260805 22:23:43.318761 31779 raft_consensus.cc:348] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.318997 31768 raft_consensus.cc:348] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.322600 31768 raft_consensus.cc:374] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.322877 31768 raft_consensus.cc:729] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.323503 31768 consensus_queue.cc:260] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.325650 31768 ts_tablet_manager.cc:1428] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260805 22:23:43.325989 31779 raft_consensus.cc:374] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.327200 31779 raft_consensus.cc:729] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.327839 31779 consensus_queue.cc:260] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.329900 31779 ts_tablet_manager.cc:1428] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.015s user 0.002s sys 0.004s
I20260805 22:23:43.326903 31768 tablet_bootstrap.cc:492] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:43.335724 31768 tablet_bootstrap.cc:654] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.338289 31779 tablet_bootstrap.cc:492] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:43.339385 31771 tablet_bootstrap.cc:492] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:43.339846 31771 ts_tablet_manager.cc:1397] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.131s user 0.010s sys 0.001s
I20260805 22:23:43.342248 31771 raft_consensus.cc:348] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.342938 31771 raft_consensus.cc:374] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.343186 31771 raft_consensus.cc:729] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.344893 31771 consensus_queue.cc:260] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.346968 31771 ts_tablet_manager.cc:1428] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260805 22:23:43.345955 31779 tablet_bootstrap.cc:654] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.349867 31771 tablet_bootstrap.cc:492] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:43.356709 31847 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:43.357214 31847 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.358870 31771 tablet_bootstrap.cc:654] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.363422 31847 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.364780 31768 tablet_bootstrap.cc:492] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:43.365222 31768 ts_tablet_manager.cc:1397] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.002s
I20260805 22:23:43.365231 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:43.365756 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:43.368353 31768 raft_consensus.cc:348] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.369041 31768 raft_consensus.cc:374] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.369297 31768 raft_consensus.cc:729] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.370007 31768 consensus_queue.cc:260] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:43.370048 31444 leader_election.cc:341] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:43.371088 31444 leader_election.cc:341] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:43.371352 31444 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:43.371920 31847 raft_consensus.cc:2726] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:43.372064 31768 ts_tablet_manager.cc:1428] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260805 22:23:43.372979 31768 tablet_bootstrap.cc:492] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:43.377435 31779 tablet_bootstrap.cc:492] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:43.377971 31779 ts_tablet_manager.cc:1397] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.040s user 0.015s sys 0.006s
I20260805 22:23:43.381179 31779 raft_consensus.cc:348] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.381999 31779 raft_consensus.cc:374] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.382294 31779 raft_consensus.cc:729] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.383061 31779 consensus_queue.cc:260] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.384311 31768 tablet_bootstrap.cc:654] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.387311 31779 ts_tablet_manager.cc:1428] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260805 22:23:43.388574 31779 tablet_bootstrap.cc:492] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:43.394737 31771 tablet_bootstrap.cc:492] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:43.395314 31771 ts_tablet_manager.cc:1397] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.046s user 0.020s sys 0.002s
I20260805 22:23:43.396895 31768 tablet_bootstrap.cc:492] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:43.397399 31768 ts_tablet_manager.cc:1397] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.025s user 0.009s sys 0.006s
I20260805 22:23:43.399813 31779 tablet_bootstrap.cc:654] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.399750 31768 raft_consensus.cc:348] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.400449 31768 raft_consensus.cc:374] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.400713 31768 raft_consensus.cc:729] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.401289 31768 consensus_queue.cc:260] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.401625 31845 consensus_queue.cc:1035] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:43.403419 31768 ts_tablet_manager.cc:1428] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260805 22:23:43.407943 31771 raft_consensus.cc:348] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.408730 31771 raft_consensus.cc:374] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.408978 31771 raft_consensus.cc:729] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.409592 31771 consensus_queue.cc:260] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.411628 31771 ts_tablet_manager.cc:1428] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.016s user 0.000s sys 0.006s
I20260805 22:23:43.412472 31771 tablet_bootstrap.cc:492] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:43.416275 31768 tablet_bootstrap.cc:492] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:43.421627 31768 tablet_bootstrap.cc:654] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.426167 31779 tablet_bootstrap.cc:492] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:43.426630 31779 ts_tablet_manager.cc:1397] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.003s
I20260805 22:23:43.427933 31786 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:43.428195 31771 tablet_bootstrap.cc:654] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.428442 31786 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.428999 31779 raft_consensus.cc:348] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.429751 31779 raft_consensus.cc:374] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.430004 31779 raft_consensus.cc:729] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.430656 31779 consensus_queue.cc:260] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.431617 31920 consensus_queue.cc:1035] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:43.432682 31779 ts_tablet_manager.cc:1428] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260805 22:23:43.433560 31779 tablet_bootstrap.cc:492] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:43.437656 31845 consensus_queue.cc:1035] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:43.446748 31786 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.449004 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:43.452437 31444 leader_election.cc:341] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:43.450510 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:43.453318 31609 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
W20260805 22:23:43.455221 31423 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260805 22:23:43.456597 31779 tablet_bootstrap.cc:654] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.456282 31444 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:43.460469 31845 consensus_queue.cc:1035] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:43.464991 31847 raft_consensus.cc:2781] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:43.465363 31847 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:43.465749 31847 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.483856 31944 consensus_queue.cc:1035] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:43.502229 31847 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.504263 31847 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.514954 31779 tablet_bootstrap.cc:492] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:43.515487 31779 ts_tablet_manager.cc:1397] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.082s user 0.009s sys 0.005s
I20260805 22:23:43.516850 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:23:43.518491 31444 leader_election.cc:341] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:43.517910 31779 raft_consensus.cc:348] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.520622 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:43.521272 31602 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.521474 31711 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.519827 31779 raft_consensus.cc:374] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.529309 31779 raft_consensus.cc:729] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.530030 31779 consensus_queue.cc:260] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.535109 31779 ts_tablet_manager.cc:1428] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20260805 22:23:43.535991 31779 tablet_bootstrap.cc:492] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:43.543051 31602 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:43.545538 31444 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:43.546694 31847 raft_consensus.cc:2781] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:43.547207 31847 raft_consensus.cc:686] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:43.548055 31847 consensus_queue.cc:237] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.584862 31779 tablet_bootstrap.cc:654] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.588315 31768 tablet_bootstrap.cc:492] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:43.588851 31768 ts_tablet_manager.cc:1397] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.173s user 0.017s sys 0.051s
I20260805 22:23:43.592401 31768 raft_consensus.cc:348] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.593182 31768 raft_consensus.cc:374] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.593483 31768 raft_consensus.cc:729] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.594098 31768 consensus_queue.cc:260] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.596309 31768 ts_tablet_manager.cc:1428] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
I20260805 22:23:43.596990 31771 tablet_bootstrap.cc:492] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:43.601369 31771 ts_tablet_manager.cc:1397] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.189s user 0.021s sys 0.058s
I20260805 22:23:43.604228 31771 raft_consensus.cc:348] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.605724 31771 raft_consensus.cc:374] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.606081 31771 raft_consensus.cc:729] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.607097 31771 consensus_queue.cc:260] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.613161 31771 ts_tablet_manager.cc:1428] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.011s user 0.005s sys 0.001s
I20260805 22:23:43.616077 31771 tablet_bootstrap.cc:492] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:43.597204 31768 tablet_bootstrap.cc:492] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:43.631166 31779 tablet_bootstrap.cc:492] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:43.631731 31779 ts_tablet_manager.cc:1397] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.096s user 0.016s sys 0.008s
I20260805 22:23:43.632723 31771 tablet_bootstrap.cc:654] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.634688 31357 catalog_manager.cc:5495] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:43.635324 31779 raft_consensus.cc:348] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.636303 31779 raft_consensus.cc:374] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.636587 31779 raft_consensus.cc:729] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.637354 31768 tablet_bootstrap.cc:654] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.637501 31779 consensus_queue.cc:260] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.657730 31845 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:43.658236 31845 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.661119 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:23:43.662497 31554 leader_election.cc:341] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:43.663300 31799 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:43.663841 31799 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.679591 31771 tablet_bootstrap.cc:492] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:43.680157 31771 ts_tablet_manager.cc:1397] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.003s
I20260805 22:23:43.679379 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:43.684587 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:43.686158 31667 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:43.684624 31771 raft_consensus.cc:348] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.686760 31771 raft_consensus.cc:374] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.687034 31865 raft_consensus.cc:2726] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:43.687703 31799 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:43.687232 31771 raft_consensus.cc:729] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.691430 31771 consensus_queue.cc:260] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.693596 31988 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:43.694054 31988 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.694813 31771 ts_tablet_manager.cc:1428] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.014s user 0.005s sys 0.001s
I20260805 22:23:43.696086 31988 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.697178 31845 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.701251 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:43.701933 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:43.705510 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:43.706673 31771 tablet_bootstrap.cc:492] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:43.706471 31847 consensus_queue.cc:1035] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260805 22:23:43.709656 31554 leader_election.cc:341] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:43.710322 31554 leader_election.cc:341] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:43.716981 31771 tablet_bootstrap.cc:654] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.718343 31779 ts_tablet_manager.cc:1428] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.086s user 0.014s sys 0.007s
I20260805 22:23:43.719436 31779 tablet_bootstrap.cc:492] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:43.717926 31554 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:43.722183 31990 raft_consensus.cc:2726] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260805 22:23:43.724278 31554 leader_election.cc:341] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:43.724726 31554 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:43.725478 31990 raft_consensus.cc:2726] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:43.726809 31779 tablet_bootstrap.cc:654] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.730716 31768 tablet_bootstrap.cc:492] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:43.730831 31711 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.731568 31768 ts_tablet_manager.cc:1397] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.135s user 0.034s sys 0.028s
I20260805 22:23:43.734251 31768 raft_consensus.cc:348] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.734722 31768 raft_consensus.cc:374] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.734946 31768 raft_consensus.cc:729] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.735423 31768 consensus_queue.cc:260] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.739737 31988 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:43.740202 31988 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.740978 31768 ts_tablet_manager.cc:1428] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260805 22:23:43.742173 31768 tablet_bootstrap.cc:492] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:43.744705 31988 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.749679 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:43.750746 31771 tablet_bootstrap.cc:492] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:43.751168 31771 ts_tablet_manager.cc:1397] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.045s user 0.013s sys 0.005s
W20260805 22:23:43.751971 31554 leader_election.cc:341] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:43.754026 31988 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:43.754886 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:43.755555 31500 raft_consensus.cc:2454] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:43.754482 31988 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.757583 31988 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.759253 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:43.759955 31500 raft_consensus.cc:2454] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:43.762509 31554 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:43.763391 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:43.763741 31554 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:43.764230 31768 tablet_bootstrap.cc:654] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.764601 31990 raft_consensus.cc:2781] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260805 22:23:43.764727 31554 leader_election.cc:341] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:43.765245 31799 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:43.767501 31988 raft_consensus.cc:2781] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:23:43.770118 31988 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:43.770458 31988 raft_consensus.cc:3037] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:43.767315 31990 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:43.767931 31799 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.773361 31799 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:43.771394 31990 raft_consensus.cc:3037] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.775914 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:43.784484 31990 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.787986 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:43.787935 31771 raft_consensus.cc:348] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:43.789577 31554 leader_election.cc:341] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:43.790176 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:43.790776 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:43.793922 31500 raft_consensus.cc:3037] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.795017 31771 raft_consensus.cc:374] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.795219 31667 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:43.796221 31771 raft_consensus.cc:729] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.790594 31988 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.792652 31990 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.801885 31799 raft_consensus.cc:2726] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:43.803175 31997 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:43.798107 31771 consensus_queue.cc:260] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.803628 31997 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.806213 31997 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.805904 31500 raft_consensus.cc:2454] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:43.808970 31554 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:43.811342 31990 raft_consensus.cc:2781] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:43.812844 31990 raft_consensus.cc:686] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:43.813026 31771 ts_tablet_manager.cc:1428] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.062s user 0.005s sys 0.004s
I20260805 22:23:43.814024 31771 tablet_bootstrap.cc:492] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:43.813990 31990 consensus_queue.cc:237] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:43.810987 31444 consensus_peers.cc:487] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 -> Peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Couldn't send request to peer feb5e7ab4b1541beaf2674c3c5500385. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:43.819980 31779 tablet_bootstrap.cc:492] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:43.820441 31779 ts_tablet_manager.cc:1397] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.101s user 0.011s sys 0.011s
I20260805 22:23:43.821161 31988 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.822666 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:43.822822 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:43.823307 31602 raft_consensus.cc:2454] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:43.823521 31500 raft_consensus.cc:3037] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:43.824484 31779 raft_consensus.cc:348] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.825169 31779 raft_consensus.cc:374] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.827600 31779 raft_consensus.cc:729] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.825989 31444 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:43.828207 31779 consensus_queue.cc:260] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.827404 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:43.830703 31997 raft_consensus.cc:2781] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:43.831046 31997 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:43.831172 31714 raft_consensus.cc:2454] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:43.831401 31997 raft_consensus.cc:3037] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.832374 31779 ts_tablet_manager.cc:1428] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.012s user 0.004s sys 0.000s
I20260805 22:23:43.836396 31944 consensus_queue.cc:1035] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:43.837909 31713 raft_consensus.cc:3037] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.838464 31779 tablet_bootstrap.cc:492] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:43.826720 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:43.841691 31786 consensus_queue.cc:1035] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:43.858929 31786 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:43.859509 31786 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:43.862012 31554 leader_election.cc:341] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:43.862677 31990 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:43.863175 31990 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.863936 31786 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.867262 31768 tablet_bootstrap.cc:492] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:43.867631 31768 ts_tablet_manager.cc:1397] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.126s user 0.016s sys 0.016s
I20260805 22:23:43.868283 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:43.868930 31501 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:43.868809 31357 catalog_manager.cc:5495] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:43.870190 31554 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:43.873907 31988 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:43.874279 31988 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:43.874586 31988 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.875912 31779 tablet_bootstrap.cc:654] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.872588 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:43.877854 31609 raft_consensus.cc:2454] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:43.876726 31997 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.886757 31988 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.892504 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:43.893131 31501 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.894800 31988 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.896596 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:23:43.899379 31554 leader_election.cc:341] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:43.893589 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:43.901767 31990 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.934682 31768 raft_consensus.cc:348] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.936525 31768 raft_consensus.cc:374] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:43.936923 31768 raft_consensus.cc:729] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:43.938060 31768 consensus_queue.cc:260] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.936237 31988 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:43.947625 31771 tablet_bootstrap.cc:654] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:43.942627 31801 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:43.948139 31801 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.938176 31444 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:43.947183 31988 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.942626 31799 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:43.950863 31799 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.951687 31988 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.943322 31786 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:43.952243 31786 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.952677 31799 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:43.953265 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:43.953967 31499 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:43.954087 31786 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.936885 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:43.954269 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:43.955286 31944 raft_consensus.cc:2781] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:43.955629 31944 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:43.955947 31944 raft_consensus.cc:3037] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.958498 31997 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:23:43.960548 31554 leader_election.cc:341] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:43.963255 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:43.963847 31713 raft_consensus.cc:3037] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.964603 31501 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
W20260805 22:23:43.965097 31554 leader_election.cc:341] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:43.965312 31768 ts_tablet_manager.cc:1428] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.097s user 0.014s sys 0.014s
I20260805 22:23:43.965550 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:43.966593 31768 tablet_bootstrap.cc:492] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:43.935477 31714 raft_consensus.cc:2454] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:43.967307 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:43.950137 31801 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:43.967926 31603 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:43.967401 31554 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:43.968837 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:43.969429 31609 raft_consensus.cc:3037] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.969849 31554 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:43.970639 31845 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:43.970686 31713 raft_consensus.cc:2454] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:43.971092 31845 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:43.971501 31845 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.974387 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:43.975267 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:43.975592 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:43.976143 31602 raft_consensus.cc:2379] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 1: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:23:43.978071 31667 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:43.980119 31799 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:43.981467 31988 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:43.982019 31988 raft_consensus.cc:686] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:43.989723 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:43.992044 31799 raft_consensus.cc:2726] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:43.982862 31988 consensus_queue.cc:237] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.998996 31779 tablet_bootstrap.cc:492] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:43.992643 31845 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:43.999387 31779 ts_tablet_manager.cc:1397] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.161s user 0.023s sys 0.019s
I20260805 22:23:44.004350 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:44.004951 31501 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.006778 31845 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.007452 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:23:44.008955 31554 leader_election.cc:341] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.009620 31768 tablet_bootstrap.cc:654] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.010272 31771 tablet_bootstrap.cc:492] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:44.011300 31771 ts_tablet_manager.cc:1397] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.197s user 0.016s sys 0.012s
I20260805 22:23:44.011219 31779 raft_consensus.cc:348] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.015024 31779 raft_consensus.cc:374] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.016288 31501 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:44.016834 31500 raft_consensus.cc:2454] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:23:44.021164 31779 raft_consensus.cc:729] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.007541 31609 raft_consensus.cc:2454] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:44.006004 31944 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.023362 31944 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.024045 31768 tablet_bootstrap.cc:492] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:44.024394 31768 ts_tablet_manager.cc:1397] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.058s user 0.006s sys 0.003s
I20260805 22:23:44.026429 31768 raft_consensus.cc:348] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.027092 31768 raft_consensus.cc:374] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.027333 31768 raft_consensus.cc:729] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.027907 31768 consensus_queue.cc:260] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.029790 31768 ts_tablet_manager.cc:1428] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20260805 22:23:44.030710 31768 tablet_bootstrap.cc:492] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:44.034150 31554 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:44.035472 31554 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:44.036043 31768 tablet_bootstrap.cc:654] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.022316 31779 consensus_queue.cc:260] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.018549 31771 raft_consensus.cc:348] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.037276 31771 raft_consensus.cc:374] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.037577 31771 raft_consensus.cc:729] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.038169 31771 consensus_queue.cc:260] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.039718 31771 ts_tablet_manager.cc:1428] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.028s user 0.002s sys 0.001s
I20260805 22:23:44.040624 31771 tablet_bootstrap.cc:492] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:44.036731 31988 raft_consensus.cc:2781] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:23:44.036609 31798 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:44.044801 31988 raft_consensus.cc:686] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:44.044847 31798 raft_consensus.cc:686] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:44.046779 31798 consensus_queue.cc:237] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.047914 31771 tablet_bootstrap.cc:654] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.049546 31988 consensus_queue.cc:237] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.052250 31779 ts_tablet_manager.cc:1428] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.053s user 0.005s sys 0.000s
I20260805 22:23:44.053777 31779 tablet_bootstrap.cc:492] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:44.059474 31779 tablet_bootstrap.cc:654] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
W20260805 22:23:44.018342 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 56473.2us
I20260805 22:23:44.075279 31779 tablet_bootstrap.cc:492] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:44.075768 31779 ts_tablet_manager.cc:1397] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.022s user 0.008s sys 0.005s
W20260805 22:23:44.078886 31444 leader_election.cc:341] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.079651 31444 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:44.080703 31944 raft_consensus.cc:2741] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260805 22:23:44.080819 31444 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:44.081707 31944 raft_consensus.cc:2781] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:44.082217 31944 raft_consensus.cc:686] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:44.083020 31944 consensus_queue.cc:237] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.085455 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:44.086102 31609 raft_consensus.cc:3037] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.093984 31609 raft_consensus.cc:2454] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:44.096517 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:44.097150 31711 raft_consensus.cc:3037] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
W20260805 22:23:44.104115 31444 leader_election.cc:341] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.104190 31711 raft_consensus.cc:2454] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
W20260805 22:23:44.105233 31444 leader_election.cc:341] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.105738 31444 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:44.106528 31944 raft_consensus.cc:2726] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:44.109683 31799 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:44.110188 31799 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.110497 31771 tablet_bootstrap.cc:492] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:44.111120 31771 ts_tablet_manager.cc:1397] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.071s user 0.019s sys 0.001s
I20260805 22:23:44.112141 31799 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:44.113270 31779 raft_consensus.cc:348] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.113988 31779 raft_consensus.cc:374] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.113132 31358 catalog_manager.cc:5495] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:44.114264 31779 raft_consensus.cc:729] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.114430 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:44.114950 31779 consensus_queue.cc:260] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.117014 31779 ts_tablet_manager.cc:1428] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.041s user 0.004s sys 0.001s
I20260805 22:23:44.123936 31771 raft_consensus.cc:348] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.124665 31771 raft_consensus.cc:374] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.124913 31771 raft_consensus.cc:729] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.125599 31771 consensus_queue.cc:260] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.118252 31444 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:44.128800 31771 ts_tablet_manager.cc:1428] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260805 22:23:44.129729 31771 tablet_bootstrap.cc:492] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:44.133509 31357 catalog_manager.cc:5495] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:44.134019 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:44.146489 31667 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:44.147675 31799 raft_consensus.cc:2726] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:44.135053 31771 tablet_bootstrap.cc:654] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.135411 32010 raft_consensus.cc:2781] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:44.148659 32010 raft_consensus.cc:686] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:44.149569 32010 consensus_queue.cc:237] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.158627 31988 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:44.159197 31988 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.161875 31771 tablet_bootstrap.cc:492] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:44.162019 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:44.162353 31771 ts_tablet_manager.cc:1397] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.033s user 0.009s sys 0.000s
W20260805 22:23:44.163236 31554 leader_election.cc:341] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.164779 31771 raft_consensus.cc:348] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.140633 31779 tablet_bootstrap.cc:492] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:44.170957 31988 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.172444 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:44.141913 31997 consensus_queue.cc:1035] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:44.170207 31799 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:44.173369 31799 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.175477 31799 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:44.177305 31779 tablet_bootstrap.cc:654] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.177983 31771 raft_consensus.cc:374] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.178743 31771 raft_consensus.cc:729] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.180496 31944 consensus_queue.cc:1035] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:44.182070 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:44.180960 31771 consensus_queue.cc:260] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.184799 31708 raft_consensus.cc:3037] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.184684 31771 ts_tablet_manager.cc:1428] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260805 22:23:44.188472 31771 tablet_bootstrap.cc:492] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:44.190997 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:44.193334 31667 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:44.194545 31799 raft_consensus.cc:2726] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260805 22:23:44.195508 31554 leader_election.cc:341] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.195984 31554 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:44.198329 31988 raft_consensus.cc:2726] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:44.198555 31771 tablet_bootstrap.cc:654] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.206197 31711 raft_consensus.cc:3037] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.208171 31798 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:44.208662 31798 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.210673 31798 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.212015 31779 tablet_bootstrap.cc:492] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:44.212457 31779 ts_tablet_manager.cc:1397] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.095s user 0.008s sys 0.008s
I20260805 22:23:44.212455 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:44.213999 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:44.215049 31779 raft_consensus.cc:348] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:44.215400 31554 leader_election.cc:341] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.215759 31779 raft_consensus.cc:374] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.216027 31779 raft_consensus.cc:729] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
W20260805 22:23:44.216089 31554 leader_election.cc:341] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.216629 31554 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:44.216706 31779 consensus_queue.cc:260] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.217751 31798 raft_consensus.cc:2726] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:44.136391 31768 tablet_bootstrap.cc:492] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:44.219051 31779 ts_tablet_manager.cc:1428] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260805 22:23:44.219738 31768 ts_tablet_manager.cc:1397] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.189s user 0.028s sys 0.015s
I20260805 22:23:44.220348 31779 tablet_bootstrap.cc:492] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:44.219390 31359 catalog_manager.cc:5495] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 2, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 2 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:44.225013 31359 catalog_manager.cc:5495] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:44.226608 31779 tablet_bootstrap.cc:654] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.235320 31768 raft_consensus.cc:348] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.243527 31768 raft_consensus.cc:374] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.243832 31768 raft_consensus.cc:729] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.244480 31768 consensus_queue.cc:260] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.246654 31768 ts_tablet_manager.cc:1428] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20260805 22:23:44.247562 31768 tablet_bootstrap.cc:492] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:44.256557 31771 tablet_bootstrap.cc:492] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:44.257066 31771 ts_tablet_manager.cc:1397] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.070s user 0.019s sys 0.007s
I20260805 22:23:44.258270 31768 tablet_bootstrap.cc:654] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.258991 32010 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:44.259388 31771 raft_consensus.cc:348] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.259470 32010 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.260171 31771 raft_consensus.cc:374] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.260437 31771 raft_consensus.cc:729] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.261477 32010 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.262601 31357 catalog_manager.cc:5495] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:44.263351 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:44.264119 31604 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:44.265841 31444 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:44.266698 32010 raft_consensus.cc:2781] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:44.267043 32010 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:44.267351 32010 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.270820 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:44.274715 32010 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.277422 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:44.277980 31604 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
W20260805 22:23:44.279085 31444 leader_election.cc:341] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.286623 31771 consensus_queue.cc:260] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.303586 32010 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.284564 31604 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:44.305739 31444 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:44.307528 32010 raft_consensus.cc:2781] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:44.308069 32010 raft_consensus.cc:686] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:44.308969 32010 consensus_queue.cc:237] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.309752 31997 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:44.310343 31997 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.312616 31997 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.314386 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:44.315114 31604 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:44.316063 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:44.316192 31444 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:44.317162 31997 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:44.317901 31997 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:44.318310 31997 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
W20260805 22:23:44.319604 31444 leader_election.cc:341] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.325915 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:23:44.328956 31444 leader_election.cc:341] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:44.330276 31554 consensus_peers.cc:487] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c -> Peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Couldn't send request to peer feb5e7ab4b1541beaf2674c3c5500385. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:44.331874 31771 ts_tablet_manager.cc:1428] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.075s user 0.007s sys 0.011s
I20260805 22:23:44.332949 31771 tablet_bootstrap.cc:492] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:44.351259 31779 tablet_bootstrap.cc:492] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:44.351418 31771 tablet_bootstrap.cc:654] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.351768 31779 ts_tablet_manager.cc:1397] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.132s user 0.009s sys 0.001s
I20260805 22:23:44.354261 31779 raft_consensus.cc:348] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.354909 31779 raft_consensus.cc:374] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.355168 31779 raft_consensus.cc:729] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.355751 31779 consensus_queue.cc:260] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.357769 31779 ts_tablet_manager.cc:1428] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260805 22:23:44.359273 31779 tablet_bootstrap.cc:492] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:44.364748 31779 tablet_bootstrap.cc:654] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.370699 31771 tablet_bootstrap.cc:492] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:44.371156 31771 ts_tablet_manager.cc:1397] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.012s
I20260805 22:23:44.374236 31779 tablet_bootstrap.cc:492] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:44.374532 31771 raft_consensus.cc:348] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.375272 31771 raft_consensus.cc:374] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.359123 31997 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.374702 31779 ts_tablet_manager.cc:1397] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.001s
I20260805 22:23:44.378657 31779 raft_consensus.cc:348] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.379333 31779 raft_consensus.cc:374] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.379580 31779 raft_consensus.cc:729] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.375522 31771 raft_consensus.cc:729] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.380168 31779 consensus_queue.cc:260] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.380718 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:44.381276 31604 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.381222 31771 consensus_queue.cc:260] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.382792 31779 ts_tablet_manager.cc:1428] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260805 22:23:44.383656 31771 ts_tablet_manager.cc:1428] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
I20260805 22:23:44.384469 31771 tablet_bootstrap.cc:492] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:44.390129 31771 tablet_bootstrap.cc:654] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.383693 31779 tablet_bootstrap.cc:492] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:44.393784 31604 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:44.395198 31444 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:44.396063 32010 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:44.396623 32010 raft_consensus.cc:686] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:44.396102 31359 catalog_manager.cc:5495] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:44.397500 32010 consensus_queue.cc:237] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.399639 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:44.400218 31997 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.397276 31779 tablet_bootstrap.cc:654] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
W20260805 22:23:44.405134 31444 leader_election.cc:341] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.409160 31711 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.410909 31768 tablet_bootstrap.cc:492] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:44.411496 31768 ts_tablet_manager.cc:1397] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.164s user 0.051s sys 0.069s
I20260805 22:23:44.412065 31771 tablet_bootstrap.cc:492] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:44.412485 31771 ts_tablet_manager.cc:1397] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.006s
I20260805 22:23:44.414120 31768 raft_consensus.cc:348] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.415925 31768 raft_consensus.cc:374] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.417183 31768 raft_consensus.cc:729] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.417868 31779 tablet_bootstrap.cc:492] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:44.418322 31779 ts_tablet_manager.cc:1397] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.003s
I20260805 22:23:44.420579 31779 raft_consensus.cc:348] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.421082 31779 raft_consensus.cc:374] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.421277 31779 raft_consensus.cc:729] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.421849 31779 consensus_queue.cc:260] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.423350 31779 ts_tablet_manager.cc:1428] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260805 22:23:44.423995 31779 tablet_bootstrap.cc:492] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:44.424279 31356 catalog_manager.cc:5495] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:44.417894 31768 consensus_queue.cc:260] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.414793 31771 raft_consensus.cc:348] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.428617 31771 raft_consensus.cc:374] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.428884 31771 raft_consensus.cc:729] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.429883 31768 ts_tablet_manager.cc:1428] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260805 22:23:44.427750 31944 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:44.429477 31771 consensus_queue.cc:260] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.430313 31944 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.431516 31771 ts_tablet_manager.cc:1428] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260805 22:23:44.432163 31944 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.432330 31771 tablet_bootstrap.cc:492] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:44.433693 31768 tablet_bootstrap.cc:492] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:44.426604 31798 consensus_queue.cc:1035] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:44.437282 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:44.439911 31771 tablet_bootstrap.cc:654] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.445050 31779 tablet_bootstrap.cc:654] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
W20260805 22:23:44.445015 31554 consensus_peers.cc:487] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c -> Peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Couldn't send request to peer feb5e7ab4b1541beaf2674c3c5500385. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:44.446362 31988 consensus_queue.cc:1035] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:44.447590 31768 tablet_bootstrap.cc:654] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.455229 31771 tablet_bootstrap.cc:492] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:44.455641 31771 ts_tablet_manager.cc:1397] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.000s
I20260805 22:23:44.458101 31713 raft_consensus.cc:2454] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:44.460242 31444 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:44.460862 31779 tablet_bootstrap.cc:492] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:44.457878 31771 raft_consensus.cc:348] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.460985 32010 raft_consensus.cc:2781] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:44.461432 32010 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:44.461824 32010 raft_consensus.cc:3037] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.461361 31779 ts_tablet_manager.cc:1397] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.037s user 0.011s sys 0.002s
I20260805 22:23:44.467947 31779 raft_consensus.cc:348] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.468677 31779 raft_consensus.cc:374] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.468945 31779 raft_consensus.cc:729] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.469676 31501 raft_consensus.cc:1261] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260805 22:23:44.469563 31779 consensus_queue.cc:260] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.471340 31798 consensus_queue.cc:1035] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:44.472143 31779 ts_tablet_manager.cc:1428] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260805 22:23:44.472493 31771 raft_consensus.cc:374] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.472779 31771 raft_consensus.cc:729] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.470118 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:44.473579 31604 raft_consensus.cc:2454] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:44.474318 31771 consensus_queue.cc:260] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.479251 31771 ts_tablet_manager.cc:1428] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.023s user 0.006s sys 0.000s
I20260805 22:23:44.480149 31771 tablet_bootstrap.cc:492] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:44.480058 31713 raft_consensus.cc:3037] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 2
I20260805 22:23:44.475605 31779 tablet_bootstrap.cc:492] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:44.484323 31798 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:44.484793 31798 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.486747 31798 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.488535 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:44.488528 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:44.490192 31554 leader_election.cc:341] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:44.490828 31554 leader_election.cc:341] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.491194 31554 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:44.492825 31988 raft_consensus.cc:2726] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:44.491294 31713 raft_consensus.cc:1261] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260805 22:23:44.496797 31771 tablet_bootstrap.cc:654] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.499562 31779 tablet_bootstrap.cc:654] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.500155 31988 consensus_queue.cc:1035] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260805 22:23:44.500774 31423 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260805 22:23:44.483850 32010 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.503149 32010 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.507037 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:44.507695 31714 raft_consensus.cc:3037] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.508891 31768 tablet_bootstrap.cc:492] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:44.509323 31768 ts_tablet_manager.cc:1397] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.076s user 0.019s sys 0.000s
I20260805 22:23:44.510334 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:44.511009 31604 raft_consensus.cc:3037] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.512534 31768 raft_consensus.cc:348] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.513209 31768 raft_consensus.cc:374] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.513475 31768 raft_consensus.cc:729] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.514132 31786 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:44.514093 31768 consensus_queue.cc:260] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.514631 31786 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.517221 31786 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.531368 31768 ts_tablet_manager.cc:1428] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260805 22:23:44.532419 31768 tablet_bootstrap.cc:492] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:44.545431 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:44.557083 31786 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:44.557641 31786 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.559669 31786 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.561674 31845 consensus_queue.cc:1035] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:44.563113 31714 raft_consensus.cc:2454] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
W20260805 22:23:44.565274 31444 leader_election.cc:341] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.566968 31990 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:44.567467 31990 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.568317 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:44.568944 31602 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:44.569370 31990 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.554291 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:44.570516 31444 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:23:44.571728 31444 leader_election.cc:341] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.572139 31444 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:44.570142 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:44.573875 31997 raft_consensus.cc:2781] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:44.575711 31997 raft_consensus.cc:686] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:44.573398 31604 raft_consensus.cc:2454] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:44.574992 32010 raft_consensus.cc:2726] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:44.580750 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:44.581463 31779 tablet_bootstrap.cc:492] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:44.581419 31709 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:44.582048 31779 ts_tablet_manager.cc:1397] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.107s user 0.016s sys 0.004s
I20260805 22:23:44.583093 31554 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:44.585675 31779 raft_consensus.cc:348] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.586613 31779 raft_consensus.cc:374] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.586875 31779 raft_consensus.cc:729] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.586989 31771 tablet_bootstrap.cc:492] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:44.587405 31771 ts_tablet_manager.cc:1397] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.107s user 0.017s sys 0.017s
I20260805 22:23:44.587499 31779 consensus_queue.cc:260] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.589569 31779 ts_tablet_manager.cc:1428] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260805 22:23:44.590613 31779 tablet_bootstrap.cc:492] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:44.591547 31997 consensus_queue.cc:237] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.589725 31798 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:44.589483 32010 consensus_queue.cc:1035] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
I20260805 22:23:44.594992 31798 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260805 22:23:44.596630 31444 leader_election.cc:341] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.596954 31798 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.597366 31444 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:44.598667 31786 raft_consensus.cc:2781] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:44.598994 31786 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:44.599298 31786 raft_consensus.cc:3037] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.606184 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:44.606940 31495 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:44.621659 31771 raft_consensus.cc:348] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.622650 31771 raft_consensus.cc:374] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.622982 31771 raft_consensus.cc:729] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.623617 31771 consensus_queue.cc:260] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.616015 31786 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.628239 31786 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.604007 31798 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.616598 32010 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:44.633857 31798 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.636236 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:44.636859 31711 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.643113 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:44.643782 31499 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.644054 31711 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:44.646193 31771 ts_tablet_manager.cc:1428] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.058s user 0.006s sys 0.000s
I20260805 22:23:44.647135 31771 tablet_bootstrap.cc:492] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:44.650305 31554 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:44.659718 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:44.660483 31604 raft_consensus.cc:3037] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.653687 31845 consensus_queue.cc:1035] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260805 22:23:44.665102 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:44.665841 31711 raft_consensus.cc:3037] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.652338 31988 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:44.666739 31988 raft_consensus.cc:686] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:44.667680 31988 consensus_queue.cc:237] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.683741 31499 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:44.686237 31990 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:44.686141 31944 consensus_queue.cc:1035] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:44.686861 31990 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.689030 31990 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.651651 32010 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.705600 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:44.716203 31786 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:44.716851 31786 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.718797 31711 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:44.767756 31768 tablet_bootstrap.cc:654] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.723865 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:44.768441 31801 consensus_queue.cc:1035] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:44.728266 32010 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.718956 31786 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.737884 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:23:44.774439 31554 leader_election.cc:341] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.778028 31771 tablet_bootstrap.cc:654] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.731498 31997 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:44.747257 31604 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
W20260805 22:23:44.782403 31554 leader_election.cc:341] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.782876 31554 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:44.785027 31845 raft_consensus.cc:2726] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:44.790077 31779 tablet_bootstrap.cc:654] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.796096 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:44.783705 31997 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:44.801631 31444 leader_election.cc:341] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.804979 31444 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:44.806170 31944 raft_consensus.cc:2781] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:44.806658 31944 raft_consensus.cc:686] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
W20260805 22:23:44.807884 31444 leader_election.cc:341] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.807440 31944 consensus_queue.cc:237] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.808318 31444 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:44.809114 32010 raft_consensus.cc:2726] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:44.811000 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:44.811514 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:44.820691 31997 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.822623 31358 catalog_manager.cc:5495] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
W20260805 22:23:44.822714 31444 leader_election.cc:341] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.827493 31771 tablet_bootstrap.cc:492] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:44.828083 31771 ts_tablet_manager.cc:1397] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.181s user 0.019s sys 0.001s
I20260805 22:23:44.838080 31779 tablet_bootstrap.cc:492] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:44.839150 31779 ts_tablet_manager.cc:1397] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.249s user 0.014s sys 0.008s
I20260805 22:23:44.841809 31779 raft_consensus.cc:348] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.842474 31779 raft_consensus.cc:374] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.842777 31779 raft_consensus.cc:729] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.843417 31779 consensus_queue.cc:260] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.845011 31779 ts_tablet_manager.cc:1428] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260805 22:23:44.845860 31779 tablet_bootstrap.cc:492] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:44.859546 31865 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:44.869897 31865 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.870998 31779 tablet_bootstrap.cc:654] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.871194 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:44.872433 31604 raft_consensus.cc:2454] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:44.872555 31865 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:23:44.873807 31444 leader_election.cc:341] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:44.874220 31444 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:44.876425 31444 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:44.877091 31944 raft_consensus.cc:2781] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:44.877472 31944 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:44.877799 31944 raft_consensus.cc:3037] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.881410 31786 raft_consensus.cc:2726] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:44.883086 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:44.883747 31708 raft_consensus.cc:2454] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:44.884128 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:44.884744 31603 raft_consensus.cc:2454] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:44.890285 31997 consensus_queue.cc:1035] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:44.893333 31768 tablet_bootstrap.cc:492] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:44.893810 31768 ts_tablet_manager.cc:1397] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.362s user 0.028s sys 0.018s
I20260805 22:23:44.898309 31667 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:44.887077 32010 consensus_queue.cc:1035] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
I20260805 22:23:44.874020 31801 consensus_queue.cc:1035] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.020s
I20260805 22:23:44.899163 31865 raft_consensus.cc:2781] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:44.899514 31865 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:44.899833 31865 raft_consensus.cc:3037] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.901871 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:44.903548 31771 raft_consensus.cc:348] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.904217 31771 raft_consensus.cc:374] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.904464 31771 raft_consensus.cc:729] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.905089 31771 consensus_queue.cc:260] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.912627 31771 ts_tablet_manager.cc:1428] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.084s user 0.006s sys 0.000s
I20260805 22:23:44.901041 31944 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.915450 31771 tablet_bootstrap.cc:492] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:44.916407 31499 raft_consensus.cc:2379] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 1: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:23:44.921721 31944 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:44.922657 31714 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.923215 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:44.923395 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:44.923833 31602 raft_consensus.cc:3037] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:44.930814 31768 raft_consensus.cc:348] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.931558 31768 raft_consensus.cc:374] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.931905 31602 raft_consensus.cc:2454] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:44.931828 31768 raft_consensus.cc:729] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.938066 31865 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.937211 31768 consensus_queue.cc:260] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.945798 31708 raft_consensus.cc:2379] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 1: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:23:44.947213 31444 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:44.948714 31768 ts_tablet_manager.cc:1428] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.055s user 0.004s sys 0.003s
I20260805 22:23:44.949658 31768 tablet_bootstrap.cc:492] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:44.948638 32010 raft_consensus.cc:2781] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:44.952996 32010 raft_consensus.cc:686] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:44.953855 32010 consensus_queue.cc:237] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.957329 31865 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:44.962893 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:44.958398 31944 consensus_queue.cc:1035] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260805 22:23:44.969175 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:44.970666 31602 raft_consensus.cc:2379] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 1: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:23:44.972932 31771 tablet_bootstrap.cc:654] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.974479 31667 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:44.974355 31356 catalog_manager.cc:5495] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:44.978147 31356 catalog_manager.cc:5495] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:44.979653 31779 tablet_bootstrap.cc:492] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:44.980144 31779 ts_tablet_manager.cc:1397] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.134s user 0.008s sys 0.009s
I20260805 22:23:44.983340 31779 raft_consensus.cc:348] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.984341 31779 raft_consensus.cc:374] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:44.977401 31865 raft_consensus.cc:2726] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260805 22:23:44.986770 31768 tablet_bootstrap.cc:654] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:44.984584 31779 raft_consensus.cc:729] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:44.988106 31779 consensus_queue.cc:260] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:44.990370 31779 ts_tablet_manager.cc:1428] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260805 22:23:44.992475 31779 tablet_bootstrap.cc:492] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
W20260805 22:23:44.998413 31444 consensus_peers.cc:487] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 -> Peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Couldn't send request to peer feb5e7ab4b1541beaf2674c3c5500385. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:45.015551 31779 tablet_bootstrap.cc:654] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:45.015982 31771 tablet_bootstrap.cc:492] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:45.016594 31771 ts_tablet_manager.cc:1397] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.103s user 0.018s sys 0.014s
I20260805 22:23:45.024876 31768 tablet_bootstrap.cc:492] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:45.025405 31768 ts_tablet_manager.cc:1397] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.076s user 0.006s sys 0.006s
I20260805 22:23:45.027793 31768 raft_consensus.cc:348] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.028474 31768 raft_consensus.cc:374] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:45.028735 31768 raft_consensus.cc:729] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:45.029291 31768 consensus_queue.cc:260] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.031293 31944 consensus_queue.cc:1035] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:45.034044 31779 tablet_bootstrap.cc:492] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:45.034446 31779 ts_tablet_manager.cc:1397] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.000s
I20260805 22:23:45.036178 31779 raft_consensus.cc:348] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.036656 31779 raft_consensus.cc:374] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:45.036855 31779 raft_consensus.cc:729] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:45.037338 31779 consensus_queue.cc:260] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.038859 31779 ts_tablet_manager.cc:1428] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260805 22:23:45.039546 31779 tablet_bootstrap.cc:492] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:45.043022 31768 ts_tablet_manager.cc:1428] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260805 22:23:45.044164 31768 tablet_bootstrap.cc:492] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:45.051103 31357 catalog_manager.cc:5495] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:45.052551 31768 tablet_bootstrap.cc:654] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:45.046248 31771 raft_consensus.cc:348] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.056895 31771 raft_consensus.cc:374] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:45.057488 31771 raft_consensus.cc:729] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:45.058557 31771 consensus_queue.cc:260] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.060968 31771 ts_tablet_manager.cc:1428] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.044s user 0.002s sys 0.003s
I20260805 22:23:45.062089 31771 tablet_bootstrap.cc:492] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:45.073884 31779 tablet_bootstrap.cc:654] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:45.080590 31768 tablet_bootstrap.cc:492] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:45.081110 31768 ts_tablet_manager.cc:1397] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.005s
I20260805 22:23:45.085572 31768 raft_consensus.cc:348] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.086381 31768 raft_consensus.cc:374] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:45.086663 31768 raft_consensus.cc:729] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:45.087361 31997 consensus_queue.cc:1035] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:45.088433 31771 tablet_bootstrap.cc:654] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:45.096062 31768 consensus_queue.cc:260] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.096931 31779 tablet_bootstrap.cc:492] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:45.097424 31779 ts_tablet_manager.cc:1397] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.058s user 0.012s sys 0.009s
I20260805 22:23:45.098343 31768 ts_tablet_manager.cc:1428] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260805 22:23:45.099237 31768 tablet_bootstrap.cc:492] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:45.100033 31779 raft_consensus.cc:348] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.100714 31779 raft_consensus.cc:374] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:45.100965 31779 raft_consensus.cc:729] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:45.102233 31779 consensus_queue.cc:260] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.104391 31779 ts_tablet_manager.cc:1428] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.007s user 0.003s sys 0.001s
I20260805 22:23:45.105243 31779 tablet_bootstrap.cc:492] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:45.105435 31771 tablet_bootstrap.cc:492] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:45.106828 31771 ts_tablet_manager.cc:1397] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.004s
I20260805 22:23:45.114089 31771 raft_consensus.cc:348] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.114811 31771 raft_consensus.cc:374] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:45.115072 31771 raft_consensus.cc:729] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:45.116672 31771 consensus_queue.cc:260] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.118918 31771 ts_tablet_manager.cc:1428] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.012s user 0.003s sys 0.002s
I20260805 22:23:45.120025 31768 tablet_bootstrap.cc:654] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:45.120218 31771 tablet_bootstrap.cc:492] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:45.120460 31779 tablet_bootstrap.cc:654] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:45.128939 31771 tablet_bootstrap.cc:654] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:45.132678 31997 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:45.152606 31779 tablet_bootstrap.cc:492] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:45.153838 31779 ts_tablet_manager.cc:1397] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.049s user 0.024s sys 0.001s
I20260805 22:23:45.153694 31997 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.155774 31997 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:45.156780 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:45.158897 31444 leader_election.cc:341] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:45.159116 31779 raft_consensus.cc:348] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.159866 31779 raft_consensus.cc:374] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:45.160125 31779 raft_consensus.cc:729] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:45.160257 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:45.160723 31779 consensus_queue.cc:260] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:45.162086 31444 leader_election.cc:341] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:45.162313 31779 ts_tablet_manager.cc:1428] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.008s user 0.003s sys 0.003s
I20260805 22:23:45.162886 31768 tablet_bootstrap.cc:492] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:45.163326 31768 ts_tablet_manager.cc:1397] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.064s user 0.008s sys 0.002s
I20260805 22:23:45.164218 31444 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:45.164718 32091 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:45.165163 31997 raft_consensus.cc:2726] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:45.165650 31768 raft_consensus.cc:348] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.165177 32091 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.166357 31768 raft_consensus.cc:374] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:45.166632 31768 raft_consensus.cc:729] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:45.167186 31768 consensus_queue.cc:260] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.167848 32091 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:45.168704 31768 ts_tablet_manager.cc:1428] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260805 22:23:45.169057 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:45.169324 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:45.170814 31667 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:45.171583 31865 raft_consensus.cc:2726] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:45.172806 31768 tablet_bootstrap.cc:492] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:45.174981 31779 tablet_bootstrap.cc:492] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:45.178340 31768 tablet_bootstrap.cc:654] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:45.181227 31779 tablet_bootstrap.cc:654] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:45.186476 31771 tablet_bootstrap.cc:492] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:45.187023 31771 ts_tablet_manager.cc:1397] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.067s user 0.010s sys 0.003s
I20260805 22:23:45.189913 31771 raft_consensus.cc:348] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.190900 31771 raft_consensus.cc:374] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:45.191172 31771 raft_consensus.cc:729] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:45.191839 31771 consensus_queue.cc:260] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.194617 31798 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:45.195130 31798 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.196982 31798 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:45.197928 31771 ts_tablet_manager.cc:1428] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260805 22:23:45.198836 31771 tablet_bootstrap.cc:492] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:45.199221 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:45.199956 31495 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:45.200318 31865 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:45.200786 31865 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.201666 31554 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:45.202889 31865 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:45.203393 32067 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:45.203861 32067 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:45.204221 32067 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:45.204540 31771 tablet_bootstrap.cc:654] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:45.206112 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:45.208122 31554 leader_election.cc:341] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:45.211630 31768 tablet_bootstrap.cc:492] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:45.211871 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:45.212203 31768 ts_tablet_manager.cc:1397] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.040s user 0.015s sys 0.006s
I20260805 22:23:45.214797 31779 tablet_bootstrap.cc:492] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:45.215235 31779 ts_tablet_manager.cc:1397] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.052s user 0.015s sys 0.006s
I20260805 22:23:45.217046 32067 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.217334 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:45.217641 31768 raft_consensus.cc:348] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.218284 31768 raft_consensus.cc:374] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:45.218538 31768 raft_consensus.cc:729] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:45.219350 31667 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:45.220166 31865 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:45.219142 31768 consensus_queue.cc:260] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.220678 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:45.221776 31495 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:45.222672 31768 ts_tablet_manager.cc:1428] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260805 22:23:45.223935 31768 tablet_bootstrap.cc:492] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:45.223605 31779 raft_consensus.cc:348] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.224290 31779 raft_consensus.cc:374] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:45.224557 31779 raft_consensus.cc:729] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:45.224821 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:45.225201 31779 consensus_queue.cc:260] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:45.226258 31554 leader_election.cc:341] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:45.227304 31779 ts_tablet_manager.cc:1428] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260805 22:23:45.276059 31997 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:45.235875 31495 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:45.277050 31779 tablet_bootstrap.cc:492] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:45.276692 31997 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.279925 31997 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:45.283568 31779 tablet_bootstrap.cc:654] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:45.286456 31865 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:45.286988 31865 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.289502 31865 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:45.295459 31779 tablet_bootstrap.cc:492] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:45.295990 31779 ts_tablet_manager.cc:1397] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.019s user 0.010s sys 0.006s
I20260805 22:23:45.297168 31990 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:45.297653 31990 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.298084 31779 raft_consensus.cc:348] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.298554 31779 raft_consensus.cc:374] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:45.298720 31779 raft_consensus.cc:729] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:45.299108 31779 consensus_queue.cc:260] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.299477 31990 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:45.300812 31779 ts_tablet_manager.cc:1428] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260805 22:23:45.300881 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:45.302342 31779 tablet_bootstrap.cc:492] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:45.302228 31603 raft_consensus.cc:2454] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:45.303860 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:45.306595 31554 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:45.307235 31768 tablet_bootstrap.cc:654] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:45.307773 31845 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:45.308226 31845 raft_consensus.cc:686] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
W20260805 22:23:45.305554 31444 leader_election.cc:341] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:45.312000 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:45.312630 31709 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:45.312881 31779 tablet_bootstrap.cc:654] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:45.309001 31845 consensus_queue.cc:237] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.318061 31554 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:45.318187 31444 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:45.229713 32067 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:45.321277 31798 consensus_queue.cc:1035] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:45.325773 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:45.325845 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:45.326499 31500 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:45.326612 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:45.328110 31990 consensus_queue.cc:1035] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:45.320804 31997 raft_consensus.cc:2781] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:45.331526 31997 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:45.332537 31997 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:45.330019 32067 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:45.341410 31771 tablet_bootstrap.cc:492] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:45.341943 31771 ts_tablet_manager.cc:1397] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.143s user 0.015s sys 0.006s
I20260805 22:23:45.335912 32067 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:45.346104 32067 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:45.355204 31771 raft_consensus.cc:348] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.355963 31771 raft_consensus.cc:374] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:45.356242 31771 raft_consensus.cc:729] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:45.356827 31771 consensus_queue.cc:260] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.358664 31801 consensus_queue.cc:1035] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:45.360525 31771 ts_tablet_manager.cc:1428] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260805 22:23:45.361506 31771 tablet_bootstrap.cc:492] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:45.365082 31359 catalog_manager.cc:5495] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:45.370632 32091 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:45.369925 31798 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:45.379026 31798 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.380797 31709 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:45.381068 31798 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:45.373813 31865 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:45.390714 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:45.390863 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:45.391551 31499 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:45.393641 31779 tablet_bootstrap.cc:492] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:45.393266 31865 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.401288 31779 ts_tablet_manager.cc:1397] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.099s user 0.021s sys 0.002s
I20260805 22:23:45.395001 31554 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:45.377955 32091 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.397781 31845 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:45.401762 31711 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:45.407338 31779 raft_consensus.cc:348] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.408814 31779 raft_consensus.cc:374] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:45.409199 31779 raft_consensus.cc:729] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:45.409904 31779 consensus_queue.cc:260] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.378502 31997 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.405848 31798 raft_consensus.cc:2781] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:45.405977 31845 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.429572 32091 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:45.472576 31798 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:45.456923 31801 consensus_queue.cc:1035] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.016s
I20260805 22:23:45.473281 31798 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:45.474295 31768 tablet_bootstrap.cc:492] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:45.474808 31768 ts_tablet_manager.cc:1397] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.251s user 0.020s sys 0.008s
I20260805 22:23:45.476231 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:45.477042 31602 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:45.482916 31768 raft_consensus.cc:348] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.483858 31768 raft_consensus.cc:374] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:45.484155 31768 raft_consensus.cc:729] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
W20260805 22:23:45.414937 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 43714.9us
I20260805 22:23:45.491585 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:45.492321 31500 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:45.432636 32067 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.494060 31865 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:45.494300 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:45.494832 31603 raft_consensus.cc:2379] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 1: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:23:45.495018 31495 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:45.497323 31771 tablet_bootstrap.cc:654] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
W20260805 22:23:45.500732 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 97597.8us
I20260805 22:23:45.484836 31768 consensus_queue.cc:260] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.489082 31602 raft_consensus.cc:2454] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:45.433921 31997 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:45.497466 32067 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:45.531008 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:45.561322 32081 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:45.566900 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:45.645928 31779 ts_tablet_manager.cc:1428] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.244s user 0.014s sys 0.005s
I20260805 22:23:45.498590 31845 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:45.567042 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:45.660279 31779 tablet_bootstrap.cc:492] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:45.662352 31444 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:45.537909 32079 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:45.603790 31706 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:45.652912 31495 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:45.563387 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:45.541055 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:45.569223 31667 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:45.660404 32081 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.619772 31798 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.687690 31768 ts_tablet_manager.cc:1428] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.212s user 0.004s sys 0.012s
I20260805 22:23:45.689369 31768 tablet_bootstrap.cc:492] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
W20260805 22:23:45.686156 31554 leader_election.cc:341] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:45.693580 32138 raft_consensus.cc:2781] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:45.694010 32138 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:45.682964 31714 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
W20260805 22:23:45.701643 31554 leader_election.cc:341] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:45.702008 31554 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:23:45.687959 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 118819us
I20260805 22:23:45.680104 32079 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.698570 32138 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:45.762542 32147 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:45.763057 32147 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.763510 32145 raft_consensus.cc:2781] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:45.764040 32145 raft_consensus.cc:686] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:45.765010 32147 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:45.786805 32134 raft_consensus.cc:2726] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260805 22:23:45.787082 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 82795.2us
I20260805 22:23:45.761013 32079 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:45.828642 31714 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:45.828853 32081 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:45.834903 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:45.764896 32145 consensus_queue.cc:237] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:45.838887 31554 leader_election.cc:341] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:45.785224 31667 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:45.841828 31779 tablet_bootstrap.cc:654] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:45.843346 31554 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:45.828647 31495 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:45.864006 31704 raft_consensus.cc:3037] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:45.867241 32147 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:45.873550 32148 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:45.874097 32148 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.876286 32148 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:45.880697 32133 raft_consensus.cc:2781] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:45.881031 32147 raft_consensus.cc:686] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:45.881796 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:45.882014 32147 consensus_queue.cc:237] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.881101 32133 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260805 22:23:45.866413 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 104522us
I20260805 22:23:45.886494 32133 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:45.885682 31798 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:45.922889 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:45.938341 32138 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.940531 32138 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:45.951190 31771 tablet_bootstrap.cc:492] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:45.953209 31771 ts_tablet_manager.cc:1397] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.592s user 0.027s sys 0.029s
W20260805 22:23:45.956795 31554 consensus_peers.cc:487] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c -> Peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Couldn't send request to peer feb5e7ab4b1541beaf2674c3c5500385. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:45.958153 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:45.922950 32079 consensus_queue.cc:1035] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20260805 22:23:45.949263 31602 raft_consensus.cc:2379] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 1: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:23:45.963330 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:45.971043 31771 raft_consensus.cc:348] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.971750 31771 raft_consensus.cc:374] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:45.972019 31771 raft_consensus.cc:729] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:45.972541 31771 consensus_queue.cc:260] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:45.970659 32144 consensus_queue.cc:1035] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260805 22:23:45.978830 31771 ts_tablet_manager.cc:1428] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.025s user 0.005s sys 0.001s
I20260805 22:23:46.018050 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:23:46.019436 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 91409us
I20260805 22:23:46.025297 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:46.025424 31667 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:23:46.028299 31667 leader_election.cc:341] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:46.029070 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:46.034866 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:46.035879 31494 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:46.043725 31768 tablet_bootstrap.cc:654] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.048440 32138 raft_consensus.cc:2726] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:46.020478 31492 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:45.973851 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:46.051633 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:46.051646 31713 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
W20260805 22:23:46.073351 31554 leader_election.cc:341] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:46.074215 31554 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:46.075675 32147 raft_consensus.cc:2726] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260805 22:23:46.077549 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 56445us
I20260805 22:23:46.078545 31779 tablet_bootstrap.cc:492] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:46.079968 31779 ts_tablet_manager.cc:1397] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.419s user 0.007s sys 0.008s
I20260805 22:23:46.085492 31771 tablet_bootstrap.cc:492] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:46.078456 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:46.091753 31779 raft_consensus.cc:348] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:46.093619 31444 leader_election.cc:341] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:46.100876 31779 raft_consensus.cc:374] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.109287 31779 raft_consensus.cc:729] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.114908 31779 consensus_queue.cc:260] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:46.115806 31423 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260805 22:23:46.118639 31779 ts_tablet_manager.cc:1428] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.036s user 0.004s sys 0.005s
I20260805 22:23:46.119068 31801 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:46.120249 31779 tablet_bootstrap.cc:492] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:46.119841 31801 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.122493 31801 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:46.125205 31768 tablet_bootstrap.cc:492] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:46.105360 32133 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.126302 31768 ts_tablet_manager.cc:1397] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.437s user 0.029s sys 0.026s
I20260805 22:23:46.132124 31554 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:46.133633 31494 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:46.139508 31709 raft_consensus.cc:2379] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 1: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:23:46.140388 32133 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:23:46.150698 31444 leader_election.cc:341] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:46.160542 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:46.161088 31771 tablet_bootstrap.cc:654] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.166064 31768 raft_consensus.cc:348] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.171904 31768 raft_consensus.cc:374] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.172142 31768 raft_consensus.cc:729] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.173308 32141 raft_consensus.cc:2781] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:46.173187 32131 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:23:46.173753 32141 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:46.174113 32141 raft_consensus.cc:3037] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:46.173877 32131 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.172647 31768 consensus_queue.cc:260] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.175740 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:46.178371 31358 catalog_manager.cc:5495] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:46.176705 31609 raft_consensus.cc:2379] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 1: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:23:46.180311 31768 ts_tablet_manager.cc:1428] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.045s user 0.006s sys 0.006s
I20260805 22:23:46.181713 31768 tablet_bootstrap.cc:492] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
W20260805 22:23:46.183215 31667 leader_election.cc:341] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:46.193584 32131 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:46.196753 32147 consensus_queue.cc:1035] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:46.199741 31771 tablet_bootstrap.cc:492] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:46.200176 31554 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:46.200232 31771 ts_tablet_manager.cc:1397] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.115s user 0.007s sys 0.008s
W20260805 22:23:46.201164 31444 leader_election.cc:341] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:46.201813 32147 raft_consensus.cc:2781] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:46.202574 32147 raft_consensus.cc:686] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:46.203662 32147 consensus_queue.cc:237] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.212875 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:46.213670 31492 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:46.214922 31771 raft_consensus.cc:348] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.215751 31771 raft_consensus.cc:374] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.216092 31771 raft_consensus.cc:729] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.218502 31771 consensus_queue.cc:260] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.222163 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:46.223621 31501 raft_consensus.cc:2379] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 1: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:23:46.224066 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:46.224714 31779 tablet_bootstrap.cc:654] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.226374 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:23:46.230520 31444 leader_election.cc:341] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:46.230928 31444 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:46.233894 32142 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:46.237399 31771 ts_tablet_manager.cc:1428] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.036s user 0.007s sys 0.000s
I20260805 22:23:46.240329 31667 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:46.241959 32131 raft_consensus.cc:2726] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260805 22:23:46.242182 31492 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:23:46.243039 31771 tablet_bootstrap.cc:492] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:46.243227 31667 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:46.233752 32141 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.245455 32131 raft_consensus.cc:2726] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:23:46.246789 31667 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:23:46.249889 32141 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:46.250702 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:46.251349 31706 raft_consensus.cc:3037] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
W20260805 22:23:46.251721 31444 leader_election.cc:341] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:46.252116 31444 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:46.253125 32156 raft_consensus.cc:2726] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:46.254401 32131 raft_consensus.cc:2781] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:46.254926 32131 raft_consensus.cc:686] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:23:46.255970 32131 consensus_queue.cc:237] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.257889 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:46.260689 31492 raft_consensus.cc:3037] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:46.263206 31768 tablet_bootstrap.cc:654] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.273656 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:46.274003 31706 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:46.274449 31602 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:46.275002 31554 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:46.275676 32147 raft_consensus.cc:2781] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:46.276175 32147 raft_consensus.cc:686] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:46.277138 32147 consensus_queue.cc:237] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.279173 31492 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:46.286696 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:46.290548 31779 tablet_bootstrap.cc:492] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:46.291373 31779 ts_tablet_manager.cc:1397] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.171s user 0.016s sys 0.001s
I20260805 22:23:46.293642 31444 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:46.294939 31801 raft_consensus.cc:2781] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:46.295300 31801 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:46.295574 31801 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
W20260805 22:23:46.298424 31444 leader_election.cc:341] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:46.304884 31779 raft_consensus.cc:348] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.305614 31779 raft_consensus.cc:374] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.305886 31779 raft_consensus.cc:729] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.307441 31779 consensus_queue.cc:260] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.313555 31779 ts_tablet_manager.cc:1428] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20260805 22:23:46.314548 31801 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.320111 31801 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:46.321851 32133 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:46.322434 32133 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.323786 31768 tablet_bootstrap.cc:492] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:46.324756 31768 ts_tablet_manager.cc:1397] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.143s user 0.008s sys 0.015s
I20260805 22:23:46.324679 32133 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:46.324294 31779 tablet_bootstrap.cc:492] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:46.341764 31771 tablet_bootstrap.cc:654] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.347025 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:46.347848 31602 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:46.351807 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:46.348726 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:46.353045 31711 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:46.354033 31492 raft_consensus.cc:2379] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 1: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:23:46.361419 31779 tablet_bootstrap.cc:654] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.362850 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:46.375037 31357 catalog_manager.cc:5495] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:46.378364 31768 raft_consensus.cc:348] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.379040 31768 raft_consensus.cc:374] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.380805 31667 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:46.379251 31768 raft_consensus.cc:729] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.383396 31711 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:46.383230 31768 consensus_queue.cc:260] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.383836 32131 raft_consensus.cc:2726] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:46.389127 31768 ts_tablet_manager.cc:1428] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.063s user 0.001s sys 0.006s
I20260805 22:23:46.400893 31602 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:46.411177 31768 tablet_bootstrap.cc:492] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:46.415486 31444 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:46.416272 31801 raft_consensus.cc:2781] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:46.416694 31801 raft_consensus.cc:686] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:46.430615 31768 tablet_bootstrap.cc:654] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.417467 31801 consensus_queue.cc:237] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.433228 31779 tablet_bootstrap.cc:492] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:46.433876 31779 ts_tablet_manager.cc:1397] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.111s user 0.025s sys 0.028s
I20260805 22:23:46.433992 31771 tablet_bootstrap.cc:492] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:46.434465 31771 ts_tablet_manager.cc:1397] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.192s user 0.024s sys 0.005s
I20260805 22:23:46.443444 31779 raft_consensus.cc:348] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.444149 31779 raft_consensus.cc:374] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.444401 31779 raft_consensus.cc:729] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.440248 31358 catalog_manager.cc:5495] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:46.447798 31845 consensus_queue.cc:1035] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:46.449661 32081 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:46.450102 32081 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.451825 31779 consensus_queue.cc:260] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.451884 31358 catalog_manager.cc:5495] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:46.453807 32081 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:23:46.468993 31444 consensus_peers.cc:487] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 -> Peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Couldn't send request to peer feb5e7ab4b1541beaf2674c3c5500385. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:46.470623 31771 raft_consensus.cc:348] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.473124 31771 raft_consensus.cc:374] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.473498 31771 raft_consensus.cc:729] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.474757 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:46.477787 31711 raft_consensus.cc:2454] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:46.479128 31444 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:46.479769 31801 raft_consensus.cc:2781] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:46.480160 31801 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:46.480515 31801 raft_consensus.cc:3037] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:46.456899 31779 ts_tablet_manager.cc:1428] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.023s user 0.004s sys 0.002s
I20260805 22:23:46.482436 31771 consensus_queue.cc:260] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.484287 31771 ts_tablet_manager.cc:1428] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.050s user 0.006s sys 0.000s
I20260805 22:23:46.481580 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:46.485798 31602 raft_consensus.cc:2454] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:46.486132 31771 tablet_bootstrap.cc:492] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:46.484910 31779 tablet_bootstrap.cc:492] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:46.486930 31768 tablet_bootstrap.cc:492] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:46.487337 31768 ts_tablet_manager.cc:1397] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.079s user 0.017s sys 0.003s
I20260805 22:23:46.490100 31768 raft_consensus.cc:348] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.490924 31768 raft_consensus.cc:374] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.491269 31768 raft_consensus.cc:729] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.491665 31357 catalog_manager.cc:5495] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:46.492446 31768 consensus_queue.cc:260] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.494904 31768 ts_tablet_manager.cc:1428] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260805 22:23:46.495911 31768 tablet_bootstrap.cc:492] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:46.509207 31359 catalog_manager.cc:5495] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:46.512579 31798 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:46.508862 31768 tablet_bootstrap.cc:654] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.513062 31798 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.515444 31779 tablet_bootstrap.cc:654] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.517231 31798 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:46.518869 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:46.520008 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:46.519447 31798 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:23:46.520936 31554 leader_election.cc:341] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:46.521791 31554 leader_election.cc:341] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:46.521497 31798 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.522166 31554 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:46.523069 31845 raft_consensus.cc:2726] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:46.523394 31798 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:46.525458 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:46.526319 31771 tablet_bootstrap.cc:654] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.526255 31492 raft_consensus.cc:2379] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 1: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:23:46.527876 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:46.515547 31801 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.533963 32081 consensus_queue.cc:1035] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:46.535600 31554 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:46.536307 31845 consensus_queue.cc:1035] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:46.537535 31845 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:46.549459 31801 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:46.552568 31845 raft_consensus.cc:2726] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:46.554708 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:46.555320 31711 raft_consensus.cc:3037] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:46.557529 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:46.558151 31604 raft_consensus.cc:3037] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:46.565097 31779 tablet_bootstrap.cc:492] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:46.565686 31779 ts_tablet_manager.cc:1397] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.081s user 0.015s sys 0.003s
I20260805 22:23:46.573719 31711 raft_consensus.cc:2454] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:46.574797 31444 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:46.576187 32144 raft_consensus.cc:2781] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:46.576709 32144 raft_consensus.cc:686] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:46.578749 32144 consensus_queue.cc:237] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.576167 31771 tablet_bootstrap.cc:492] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:46.586724 31779 raft_consensus.cc:348] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.587484 31779 raft_consensus.cc:374] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.587749 31779 raft_consensus.cc:729] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.590466 31771 ts_tablet_manager.cc:1397] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.105s user 0.016s sys 0.003s
I20260805 22:23:46.591034 31768 tablet_bootstrap.cc:492] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:46.591500 31768 ts_tablet_manager.cc:1397] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.096s user 0.003s sys 0.011s
I20260805 22:23:46.593972 31768 raft_consensus.cc:348] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.588362 31779 consensus_queue.cc:260] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.594552 31604 raft_consensus.cc:2454] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:46.596146 31779 ts_tablet_manager.cc:1428] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.030s user 0.004s sys 0.001s
I20260805 22:23:46.594592 31768 raft_consensus.cc:374] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.596774 31768 raft_consensus.cc:729] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.597404 31768 consensus_queue.cc:260] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.598670 31779 tablet_bootstrap.cc:492] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:46.604956 31771 raft_consensus.cc:348] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.605679 31771 raft_consensus.cc:374] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.605927 31771 raft_consensus.cc:729] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.607337 31771 consensus_queue.cc:260] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.608224 31779 tablet_bootstrap.cc:654] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.612555 31771 ts_tablet_manager.cc:1428] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20260805 22:23:46.613724 31771 tablet_bootstrap.cc:492] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:46.616746 32144 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:46.617268 32144 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.619145 31768 ts_tablet_manager.cc:1428] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.027s user 0.007s sys 0.001s
I20260805 22:23:46.621816 32144 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:46.624312 31768 tablet_bootstrap.cc:492] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:46.626351 31359 catalog_manager.cc:5495] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:46.633261 31779 tablet_bootstrap.cc:492] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:46.633769 31779 ts_tablet_manager.cc:1397] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.011s
I20260805 22:23:46.634514 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:46.631665 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:23:46.637482 31444 leader_election.cc:341] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:46.638237 31444 leader_election.cc:341] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:46.638655 31444 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:46.639768 32144 raft_consensus.cc:2726] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:46.644523 31768 tablet_bootstrap.cc:654] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.645790 32131 consensus_queue.cc:1035] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:46.646559 31779 raft_consensus.cc:348] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.647351 31779 raft_consensus.cc:374] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.648100 31779 raft_consensus.cc:729] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.649106 31771 tablet_bootstrap.cc:654] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.648782 31779 consensus_queue.cc:260] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.651011 31779 ts_tablet_manager.cc:1428] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260805 22:23:46.656601 31779 tablet_bootstrap.cc:492] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:46.662153 31779 tablet_bootstrap.cc:654] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.669287 32131 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:46.669785 32131 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.671801 32131 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:46.672873 31845 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:46.674212 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:23:46.676011 31667 leader_election.cc:341] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:46.680768 32131 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:46.682153 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:46.682284 32131 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.684427 32131 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:46.673331 31845 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.686615 32134 consensus_queue.cc:1035] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:46.687803 31845 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:46.689499 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:46.690201 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:23:46.688748 31667 leader_election.cc:341] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:46.691754 31667 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:23:46.694386 31554 leader_election.cc:341] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:46.694442 32133 raft_consensus.cc:2726] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:46.696300 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:46.697677 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:23:46.701849 31554 leader_election.cc:341] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:46.702541 31667 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:46.702217 31554 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:46.705444 32134 raft_consensus.cc:2726] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:46.703891 32133 raft_consensus.cc:2726] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:46.713686 31768 tablet_bootstrap.cc:492] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:46.714226 31768 ts_tablet_manager.cc:1397] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.093s user 0.023s sys 0.018s
I20260805 22:23:46.715695 31779 tablet_bootstrap.cc:492] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:46.716221 31779 ts_tablet_manager.cc:1397] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.060s user 0.012s sys 0.004s
I20260805 22:23:46.722714 31771 tablet_bootstrap.cc:492] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:46.723263 31771 ts_tablet_manager.cc:1397] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.110s user 0.007s sys 0.010s
I20260805 22:23:46.724933 31779 raft_consensus.cc:348] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.725714 31779 raft_consensus.cc:374] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.725960 31779 raft_consensus.cc:729] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.726593 31779 consensus_queue.cc:260] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.726922 32229 consensus_queue.cc:1035] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:46.728466 31768 raft_consensus.cc:348] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.730023 31771 raft_consensus.cc:348] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.730559 31768 raft_consensus.cc:374] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.730955 31768 raft_consensus.cc:729] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.731098 31779 ts_tablet_manager.cc:1428] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.015s user 0.003s sys 0.003s
I20260805 22:23:46.732020 31779 tablet_bootstrap.cc:492] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:46.730644 31771 raft_consensus.cc:374] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.731691 31768 consensus_queue.cc:260] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.735069 31771 raft_consensus.cc:729] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.736302 31768 ts_tablet_manager.cc:1428] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.022s user 0.004s sys 0.002s
I20260805 22:23:46.736441 31771 consensus_queue.cc:260] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.737447 31768 tablet_bootstrap.cc:492] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:46.739080 31771 ts_tablet_manager.cc:1428] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20260805 22:23:46.742545 31779 tablet_bootstrap.cc:654] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.747399 32144 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:46.749874 32144 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.753211 32144 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:46.753587 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:46.753993 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:46.754292 31706 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:46.754649 31604 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:46.755568 31444 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:46.753152 31771 tablet_bootstrap.cc:492] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:46.756873 32079 raft_consensus.cc:2781] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:46.757262 32079 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:46.757599 32079 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:46.775569 32134 consensus_queue.cc:1035] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:46.776388 31771 tablet_bootstrap.cc:654] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.779138 31768 tablet_bootstrap.cc:654] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.784245 31779 tablet_bootstrap.cc:492] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:46.784698 31779 ts_tablet_manager.cc:1397] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.053s user 0.012s sys 0.004s
I20260805 22:23:46.784505 32079 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.787328 32079 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:46.789072 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:46.789778 31602 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:46.789902 31779 raft_consensus.cc:348] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.790691 31779 raft_consensus.cc:374] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.790953 31779 raft_consensus.cc:729] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.791553 31779 consensus_queue.cc:260] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.795612 31779 ts_tablet_manager.cc:1428] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260805 22:23:46.799165 31602 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:46.800387 31768 tablet_bootstrap.cc:492] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:46.821338 31768 ts_tablet_manager.cc:1397] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.084s user 0.017s sys 0.012s
I20260805 22:23:46.800740 31771 tablet_bootstrap.cc:492] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:46.817154 31801 consensus_queue.cc:1035] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:46.823272 31771 ts_tablet_manager.cc:1397] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.070s user 0.007s sys 0.010s
I20260805 22:23:46.817966 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:46.824184 31711 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:46.824005 31768 raft_consensus.cc:348] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.824739 31768 raft_consensus.cc:374] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.825022 31768 raft_consensus.cc:729] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.825628 31444 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:46.825663 31768 consensus_queue.cc:260] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.826535 32147 consensus_queue.cc:1035] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:46.827349 31768 ts_tablet_manager.cc:1428] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20260805 22:23:46.827796 32081 raft_consensus.cc:2781] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:46.825663 31771 raft_consensus.cc:348] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.828219 31771 raft_consensus.cc:374] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.828289 32081 raft_consensus.cc:686] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:46.828491 31771 raft_consensus.cc:729] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.829104 31771 consensus_queue.cc:260] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.829149 32081 consensus_queue.cc:237] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.831173 31771 ts_tablet_manager.cc:1428] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260805 22:23:46.832105 31771 tablet_bootstrap.cc:492] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:46.820366 32134 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:46.834601 32134 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.837226 32134 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:46.837440 31771 tablet_bootstrap.cc:654] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.841192 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:46.842607 31554 leader_election.cc:341] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:46.843734 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:46.844422 31492 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:46.845432 31554 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:46.846246 31845 raft_consensus.cc:2781] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:46.846606 31845 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:46.846909 31845 raft_consensus.cc:3037] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:46.853646 31779 tablet_bootstrap.cc:492] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:46.855402 31711 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:46.859774 31779 tablet_bootstrap.cc:654] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.868794 31798 consensus_queue.cc:1035] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:46.872416 31779 tablet_bootstrap.cc:492] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:46.872898 31779 ts_tablet_manager.cc:1397] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.000s
I20260805 22:23:46.875751 31779 raft_consensus.cc:348] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.876417 31779 raft_consensus.cc:374] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:46.876677 31779 raft_consensus.cc:729] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:46.878392 31768 tablet_bootstrap.cc:492] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:46.883766 31768 tablet_bootstrap.cc:654] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:46.884079 32079 consensus_queue.cc:1035] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:46.877279 31779 consensus_queue.cc:260] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.888222 31779 ts_tablet_manager.cc:1428] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.015s user 0.007s sys 0.000s
I20260805 22:23:46.889144 31779 tablet_bootstrap.cc:492] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:46.876200 31845 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.896615 31707 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:46.918778 32229 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:46.930161 31801 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:46.930671 31801 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.937572 31801 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:46.944125 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:46.943970 32229 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.944391 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:46.944808 31711 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:46.945017 31604 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:46.946100 32229 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:46.946301 31444 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:46.947790 32156 raft_consensus.cc:2781] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:46.948119 32156 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:46.948261 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:23:46.950503 31667 leader_election.cc:341] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:46.941192 31357 catalog_manager.cc:5495] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:46.951179 32156 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:46.976768 31711 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:46.996356 32156 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:46.997918 31845 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:47.032059 32081 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:47.035019 32081 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:47.039812 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 62115.3us
I20260805 22:23:47.041155 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:47.037921 32081 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:47.042737 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:47.045195 31501 raft_consensus.cc:3037] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:47.047405 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:23:47.044528 31444 leader_election.cc:341] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:47.037986 32230 log.cc:924] Time spent T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20260805 22:23:47.049803 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:47.051117 31444 leader_election.cc:341] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:47.051488 31444 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:47.052465 32081 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:47.054929 31771 tablet_bootstrap.cc:492] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:47.055124 31779 tablet_bootstrap.cc:654] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.055349 31771 ts_tablet_manager.cc:1397] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.224s user 0.038s sys 0.041s
I20260805 22:23:47.060248 32156 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:47.039847 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:23:47.062654 31667 leader_election.cc:341] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:47.063043 31667 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:47.063994 32133 raft_consensus.cc:2726] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:47.066375 31771 raft_consensus.cc:348] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.067052 31771 raft_consensus.cc:374] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.067298 31771 raft_consensus.cc:729] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.067494 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:47.068095 31604 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:47.069321 31771 consensus_queue.cc:260] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:47.070685 31554 leader_election.cc:341] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:47.075699 31501 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:47.076340 31768 tablet_bootstrap.cc:492] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:47.076789 31768 ts_tablet_manager.cc:1397] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.199s user 0.037s sys 0.065s
I20260805 22:23:47.077961 31771 ts_tablet_manager.cc:1428] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20260805 22:23:47.078917 31771 tablet_bootstrap.cc:492] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:47.085922 31554 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:47.106684 31768 raft_consensus.cc:348] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.107481 31768 raft_consensus.cc:374] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.107822 31768 raft_consensus.cc:729] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.108678 31768 consensus_queue.cc:260] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.094929 31845 raft_consensus.cc:2781] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:47.103313 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:47.103627 32141 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:47.115303 31768 ts_tablet_manager.cc:1428] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.038s user 0.006s sys 0.002s
I20260805 22:23:47.116607 31768 tablet_bootstrap.cc:492] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:47.119911 31845 raft_consensus.cc:686] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:47.124662 31706 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:47.124475 32141 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.158533 32141 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:47.159775 31771 tablet_bootstrap.cc:654] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.162617 31768 tablet_bootstrap.cc:654] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.172696 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:47.159396 31845 consensus_queue.cc:237] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:47.176625 31554 leader_election.cc:341] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:47.179971 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:47.180658 31709 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:47.185823 31554 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:23:47.187101 32134 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:47.187489 32134 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:47.187830 32134 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:47.191345 31604 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:47.193789 32156 consensus_queue.cc:1035] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:47.196286 31779 tablet_bootstrap.cc:492] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:47.196657 31779 ts_tablet_manager.cc:1397] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.308s user 0.032s sys 0.014s
I20260805 22:23:47.205197 31444 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:47.211314 32144 raft_consensus.cc:2781] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:47.214969 32144 raft_consensus.cc:686] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:47.215853 32144 consensus_queue.cc:237] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.220794 32156 consensus_queue.cc:1035] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:47.225992 31706 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:47.238147 32252 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:47.238659 32252 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.239800 31358 catalog_manager.cc:5495] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:47.240648 32252 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:47.242908 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:47.240414 32134 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.244323 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:47.244612 31554 leader_election.cc:341] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:47.245774 32134 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:23:47.245899 31554 leader_election.cc:341] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:47.246299 31554 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:47.249681 32134 raft_consensus.cc:2726] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:47.250918 31779 raft_consensus.cc:348] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.256698 31779 raft_consensus.cc:374] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.256980 31779 raft_consensus.cc:729] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.257519 31779 consensus_queue.cc:260] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.267160 31768 tablet_bootstrap.cc:492] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:47.267657 31768 ts_tablet_manager.cc:1397] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.151s user 0.025s sys 0.006s
I20260805 22:23:47.258776 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:47.269836 31779 ts_tablet_manager.cc:1428] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.073s user 0.007s sys 0.001s
W20260805 22:23:47.270509 31554 leader_election.cc:341] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:47.268440 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:47.271910 31707 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:47.273798 31845 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:47.271924 31779 tablet_bootstrap.cc:492] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:47.274223 31845 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.282429 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:47.282423 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:47.283952 31771 tablet_bootstrap.cc:492] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:47.284386 31771 ts_tablet_manager.cc:1397] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.206s user 0.008s sys 0.017s
W20260805 22:23:47.285637 31554 leader_election.cc:341] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:47.286232 31554 leader_election.cc:341] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:47.286557 31554 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:47.287318 32134 raft_consensus.cc:2726] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:47.290825 31768 raft_consensus.cc:348] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.291499 31768 raft_consensus.cc:374] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.291766 31768 raft_consensus.cc:729] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.291641 31357 catalog_manager.cc:5495] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:47.280787 32079 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:47.293009 32079 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.292415 31768 consensus_queue.cc:260] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.280638 31845 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:47.311607 31771 raft_consensus.cc:348] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.312558 31771 raft_consensus.cc:374] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.312680 32079 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:47.314033 31768 ts_tablet_manager.cc:1428] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.046s user 0.003s sys 0.005s
I20260805 22:23:47.314358 31771 raft_consensus.cc:729] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.315096 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:47.315752 31609 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:47.317565 31771 consensus_queue.cc:260] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.318508 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:47.319128 31709 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:47.325783 31444 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:47.326599 32264 raft_consensus.cc:2781] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:47.326959 32264 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:47.327279 32264 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
W20260805 22:23:47.329075 31423 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260805 22:23:47.332566 32264 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.335107 31707 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:47.335599 31771 ts_tablet_manager.cc:1428] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.051s user 0.007s sys 0.001s
I20260805 22:23:47.338310 31771 tablet_bootstrap.cc:492] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:47.339316 31779 tablet_bootstrap.cc:654] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.336892 32144 consensus_queue.cc:1035] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:47.344846 31554 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:23:47.345705 32260 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:47.345945 32264 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:47.346180 32260 raft_consensus.cc:686] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:47.346983 32260 consensus_queue.cc:237] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.347548 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:47.348129 31709 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:47.351114 31768 tablet_bootstrap.cc:492] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:47.354091 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:47.354689 31609 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:47.355440 31709 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:47.356498 31444 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:47.357182 32264 raft_consensus.cc:2781] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:47.357676 32264 raft_consensus.cc:686] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:47.360311 31771 tablet_bootstrap.cc:654] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.360810 32264 consensus_queue.cc:237] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.370235 31609 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:47.387110 32079 consensus_queue.cc:1035] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:47.390560 31768 tablet_bootstrap.cc:654] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.399969 31359 catalog_manager.cc:5495] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:47.407778 31779 tablet_bootstrap.cc:492] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:47.408352 31779 ts_tablet_manager.cc:1397] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.137s user 0.009s sys 0.002s
I20260805 22:23:47.410630 31768 tablet_bootstrap.cc:492] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:47.411142 31768 ts_tablet_manager.cc:1397] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.013s
I20260805 22:23:47.421944 32260 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:47.422564 32260 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.427973 31359 catalog_manager.cc:5495] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:47.437503 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:23:47.440440 31554 leader_election.cc:341] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:47.440476 31768 raft_consensus.cc:348] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.439262 31779 raft_consensus.cc:348] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.441354 31768 raft_consensus.cc:374] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.441560 31779 raft_consensus.cc:374] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.444813 31779 raft_consensus.cc:729] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.441712 31768 raft_consensus.cc:729] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.445701 31779 consensus_queue.cc:260] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.445814 31768 consensus_queue.cc:260] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.447434 31768 ts_tablet_manager.cc:1428] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.036s user 0.003s sys 0.003s
I20260805 22:23:47.447889 31779 ts_tablet_manager.cc:1428] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.039s user 0.009s sys 0.002s
I20260805 22:23:47.448675 31768 tablet_bootstrap.cc:492] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:47.454650 31779 tablet_bootstrap.cc:492] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:47.454972 31768 tablet_bootstrap.cc:654] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.460836 31771 tablet_bootstrap.cc:492] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:47.461252 31771 ts_tablet_manager.cc:1397] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.123s user 0.023s sys 0.010s
I20260805 22:23:47.461699 32260 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:47.463239 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:47.469709 31554 leader_election.cc:341] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:47.470204 31554 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:47.469238 31771 raft_consensus.cc:348] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.471442 32260 raft_consensus.cc:2726] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:47.471803 31771 raft_consensus.cc:374] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.472151 31771 raft_consensus.cc:729] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.472803 31771 consensus_queue.cc:260] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.474858 31771 ts_tablet_manager.cc:1428] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260805 22:23:47.478927 31779 tablet_bootstrap.cc:654] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.482503 31771 tablet_bootstrap.cc:492] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:47.484122 32264 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:47.484580 32264 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.486510 32264 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:47.488150 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:47.491271 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:47.492126 31709 raft_consensus.cc:2379] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 1: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:23:47.494365 31444 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:47.495294 32264 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:47.501837 31771 tablet_bootstrap.cc:654] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.502125 32264 raft_consensus.cc:2726] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:47.506773 31771 tablet_bootstrap.cc:492] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:47.507094 31768 tablet_bootstrap.cc:492] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:47.507226 31771 ts_tablet_manager.cc:1397] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.025s user 0.010s sys 0.000s
I20260805 22:23:47.507730 31768 ts_tablet_manager.cc:1397] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.059s user 0.037s sys 0.021s
I20260805 22:23:47.510995 31779 tablet_bootstrap.cc:492] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:47.511446 31779 ts_tablet_manager.cc:1397] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.003s
I20260805 22:23:47.511198 31768 raft_consensus.cc:348] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.513773 31779 raft_consensus.cc:348] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.514470 31768 raft_consensus.cc:374] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.514505 31779 raft_consensus.cc:374] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.514948 31768 raft_consensus.cc:729] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.514966 31779 raft_consensus.cc:729] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.515793 31768 consensus_queue.cc:260] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.515940 31779 consensus_queue.cc:260] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.518222 31768 ts_tablet_manager.cc:1428] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.010s user 0.005s sys 0.001s
I20260805 22:23:47.518330 31779 ts_tablet_manager.cc:1428] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260805 22:23:47.516775 31771 raft_consensus.cc:348] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.519393 31771 raft_consensus.cc:374] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.519517 31779 tablet_bootstrap.cc:492] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:47.519680 31771 raft_consensus.cc:729] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.520301 31771 consensus_queue.cc:260] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.523976 31768 tablet_bootstrap.cc:492] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:47.531034 31768 tablet_bootstrap.cc:654] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.531973 31779 tablet_bootstrap.cc:654] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.533764 31771 ts_tablet_manager.cc:1428] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.026s user 0.006s sys 0.001s
I20260805 22:23:47.544373 31771 tablet_bootstrap.cc:492] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:47.547349 31779 tablet_bootstrap.cc:492] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:47.547827 31779 ts_tablet_manager.cc:1397] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.000s
I20260805 22:23:47.550077 31771 tablet_bootstrap.cc:654] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.571269 31779 raft_consensus.cc:348] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.572055 31779 raft_consensus.cc:374] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.572315 31779 raft_consensus.cc:729] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.572917 31779 consensus_queue.cc:260] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.575088 31779 ts_tablet_manager.cc:1428] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.027s user 0.012s sys 0.000s
I20260805 22:23:47.577513 31768 tablet_bootstrap.cc:492] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:47.578054 31768 ts_tablet_manager.cc:1397] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.017s
I20260805 22:23:47.578146 31771 tablet_bootstrap.cc:492] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:47.578577 31771 ts_tablet_manager.cc:1397] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.034s user 0.016s sys 0.000s
I20260805 22:23:47.580614 31768 raft_consensus.cc:348] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.580845 31771 raft_consensus.cc:348] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.581322 31768 raft_consensus.cc:374] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.581473 31771 raft_consensus.cc:374] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.581661 31768 raft_consensus.cc:729] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.581779 31771 raft_consensus.cc:729] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.582435 31771 consensus_queue.cc:260] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.582383 31768 consensus_queue.cc:260] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.583866 31779 tablet_bootstrap.cc:492] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:47.584766 31771 ts_tablet_manager.cc:1428] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260805 22:23:47.585693 31771 tablet_bootstrap.cc:492] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:47.591424 31771 tablet_bootstrap.cc:654] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.593245 31768 ts_tablet_manager.cc:1428] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260805 22:23:47.591377 31779 tablet_bootstrap.cc:654] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.611359 31768 tablet_bootstrap.cc:492] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:47.617313 31779 tablet_bootstrap.cc:492] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:47.617841 31779 ts_tablet_manager.cc:1397] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.006s
I20260805 22:23:47.619897 31768 tablet_bootstrap.cc:654] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.620201 31771 tablet_bootstrap.cc:492] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:47.620780 31771 ts_tablet_manager.cc:1397] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.008s
I20260805 22:23:47.620328 31779 raft_consensus.cc:348] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.621181 31779 raft_consensus.cc:374] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.621483 31779 raft_consensus.cc:729] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.622071 31779 consensus_queue.cc:260] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.624275 31779 ts_tablet_manager.cc:1428] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260805 22:23:47.624272 31771 raft_consensus.cc:348] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.625149 31771 raft_consensus.cc:374] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.625370 31771 raft_consensus.cc:729] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.625978 31771 consensus_queue.cc:260] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.627622 31771 ts_tablet_manager.cc:1428] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260805 22:23:47.628461 31771 tablet_bootstrap.cc:492] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:47.631685 32131 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:47.632164 32131 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.634119 32264 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:47.634208 32131 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:47.634939 31771 tablet_bootstrap.cc:654] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.634652 32264 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.636507 32264 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:47.637032 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:47.638556 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:47.639149 31709 raft_consensus.cc:2454] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:47.640676 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:47.641271 31609 raft_consensus.cc:2454] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:47.641424 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:47.642577 31501 raft_consensus.cc:2379] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 1: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:23:47.643360 31798 consensus_queue.cc:1035] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:47.644699 31779 tablet_bootstrap.cc:492] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:47.646127 31667 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:47.647382 32133 raft_consensus.cc:3037] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:47.650208 31779 tablet_bootstrap.cc:654] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.643298 31444 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:47.655820 32144 raft_consensus.cc:2781] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:47.656194 32144 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:47.656526 32144 raft_consensus.cc:3037] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:47.673840 32133 raft_consensus.cc:2726] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:47.674758 32144 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.680549 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:47.681322 31609 raft_consensus.cc:3037] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:47.693464 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:47.694216 31709 raft_consensus.cc:3037] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:47.707628 31709 raft_consensus.cc:2454] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:47.709196 31444 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:47.712002 32264 raft_consensus.cc:2781] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:47.712499 32264 raft_consensus.cc:686] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:47.713342 32264 consensus_queue.cc:237] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.764137 32144 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:47.784214 32171 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:47.796356 32133 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:47.797057 32133 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.803737 32260 consensus_queue.cc:1035] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:47.808322 31357 catalog_manager.cc:5495] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:47.821237 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:47.789755 32171 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.831274 32171 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:23:47.825616 31667 leader_election.cc:341] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:47.787120 31609 raft_consensus.cc:2454] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:47.837527 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:47.839023 31604 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:47.833456 32133 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:47.855739 31667 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:23:47.864210 31768 tablet_bootstrap.cc:492] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:47.864712 31768 ts_tablet_manager.cc:1397] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.270s user 0.024s sys 0.033s
I20260805 22:23:47.864095 31798 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:47.865746 31798 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.873838 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:47.874478 31711 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:47.880507 31798 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:47.865432 32133 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:47.881238 31768 raft_consensus.cc:348] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.881873 32133 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.882618 31768 raft_consensus.cc:374] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.882931 31768 raft_consensus.cc:729] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.883780 32133 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:47.884711 31771 tablet_bootstrap.cc:492] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:47.884757 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:47.885248 31771 ts_tablet_manager.cc:1397] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.257s user 0.028s sys 0.051s
I20260805 22:23:47.882519 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:47.885604 31768 consensus_queue.cc:260] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:47.888083 31554 leader_election.cc:341] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:47.889855 31771 raft_consensus.cc:348] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.890545 31771 raft_consensus.cc:374] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.890857 31771 raft_consensus.cc:729] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.880306 32131 raft_consensus.cc:2781] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:47.891746 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:47.892117 31771 consensus_queue.cc:260] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.893857 31768 ts_tablet_manager.cc:1428] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20260805 22:23:47.894874 31768 tablet_bootstrap.cc:492] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:47.894953 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:47.895599 31602 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:47.896677 31667 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:23:47.897104 31554 leader_election.cc:341] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:47.897495 31554 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:47.877907 32144 consensus_queue.cc:1035] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20260805 22:23:47.893941 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:47.899462 31501 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:47.892230 32131 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:47.900589 32131 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:47.900895 32260 raft_consensus.cc:2726] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:47.904888 32079 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:47.905453 32079 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.907380 32079 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:47.913096 31444 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:47.916270 32133 raft_consensus.cc:2781] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:47.916716 32133 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:47.917933 32133 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:47.917802 31771 ts_tablet_manager.cc:1428] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.032s user 0.005s sys 0.001s
I20260805 22:23:47.916340 32171 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:47.923022 32171 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:47.923380 32171 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:47.923864 32131 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.925704 32144 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:47.926189 32144 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.927719 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:47.927953 32144 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:47.928911 31771 tablet_bootstrap.cc:492] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
W20260805 22:23:47.929118 31667 leader_election.cc:341] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:47.932921 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:47.933502 31609 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:47.936923 31779 tablet_bootstrap.cc:492] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:47.939028 31779 ts_tablet_manager.cc:1397] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.294s user 0.107s sys 0.118s
I20260805 22:23:47.937464 32133 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.941326 31779 raft_consensus.cc:348] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.942054 31779 raft_consensus.cc:374] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:47.942325 31779 raft_consensus.cc:729] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:47.929415 32131 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:47.942891 31779 consensus_queue.cc:260] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.945099 31779 ts_tablet_manager.cc:1428] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260805 22:23:47.946003 31779 tablet_bootstrap.cc:492] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:47.945827 31609 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:23:47.946931 32171 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.947863 31667 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:23:47.948848 32171 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:47.948875 32131 raft_consensus.cc:2781] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:47.949671 32131 raft_consensus.cc:686] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:23:47.950553 32131 consensus_queue.cc:237] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.952186 31768 tablet_bootstrap.cc:654] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.952232 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:47.952322 31771 tablet_bootstrap.cc:654] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.957167 31501 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:47.949487 32133 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:47.949352 32171 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:47.960107 32171 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.961858 31779 tablet_bootstrap.cc:654] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:47.962145 32171 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:47.955137 31798 consensus_queue.cc:1035] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260805 22:23:47.965785 31501 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
W20260805 22:23:47.967787 31444 leader_election.cc:341] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:47.968087 31667 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:47.969622 32133 raft_consensus.cc:2781] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:47.970252 32133 raft_consensus.cc:686] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:23:47.971117 32133 consensus_queue.cc:237] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.971446 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:47.973048 31609 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:47.975050 31798 consensus_queue.cc:1035] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:47.977815 31798 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:47.978293 31798 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:47.983453 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:47.984897 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:47.980531 32302 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:47.986519 32302 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:47.987676 31554 leader_election.cc:341] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:47.989760 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:47.991209 31554 leader_election.cc:341] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:47.991550 31554 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:47.993268 32141 raft_consensus.cc:2726] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:47.994765 32302 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:47.996462 31798 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:47.997891 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:47.999828 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:48.000785 31492 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:48.003679 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:48.003247 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:48.005126 31602 raft_consensus.cc:2454] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:48.005566 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:48.022958 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:48.035151 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:48.036357 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:48.036899 31711 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:48.041489 32144 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:48.042022 32144 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.044117 32144 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:48.052613 31711 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
W20260805 22:23:48.034662 31444 leader_election.cc:341] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:48.060787 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:48.061570 31706 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:48.067107 31667 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:23:48.067914 31667 leader_election.cc:341] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:48.069096 31444 leader_election.cc:341] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:48.069815 31444 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
W20260805 22:23:48.074405 31444 leader_election.cc:341] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:48.073083 32306 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:48.078126 32306 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:48.078462 32306 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:48.073303 32144 raft_consensus.cc:2781] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:48.101540 31771 tablet_bootstrap.cc:492] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:48.073055 32171 consensus_queue.cc:1035] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20260805 22:23:48.102093 31771 ts_tablet_manager.cc:1397] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.173s user 0.015s sys 0.019s
I20260805 22:23:48.106886 31357 catalog_manager.cc:5495] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:48.110016 31357 catalog_manager.cc:5495] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:48.095458 32144 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:48.111872 32144 raft_consensus.cc:3037] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:48.148648 31768 tablet_bootstrap.cc:492] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:48.157202 31768 ts_tablet_manager.cc:1397] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.263s user 0.017s sys 0.006s
I20260805 22:23:48.157996 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:48.136540 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:48.172358 31779 tablet_bootstrap.cc:492] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:48.174028 31779 ts_tablet_manager.cc:1397] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.228s user 0.011s sys 0.051s
I20260805 22:23:48.174863 31768 raft_consensus.cc:348] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.178257 31768 raft_consensus.cc:374] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:48.178555 31768 raft_consensus.cc:729] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:48.179198 31768 consensus_queue.cc:260] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.187469 31771 raft_consensus.cc:348] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.188114 31771 raft_consensus.cc:374] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:48.188330 31771 raft_consensus.cc:729] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:48.190014 31771 consensus_queue.cc:260] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.202452 31768 ts_tablet_manager.cc:1428] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.045s user 0.007s sys 0.000s
W20260805 22:23:48.213797 31444 leader_election.cc:341] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:48.215405 32316 log.cc:924] Time spent T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.050s user 0.000s sys 0.003s
I20260805 22:23:48.217738 31771 ts_tablet_manager.cc:1428] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.115s user 0.010s sys 0.004s
I20260805 22:23:48.217834 31779 raft_consensus.cc:348] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.218680 31779 raft_consensus.cc:374] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:48.218791 31771 tablet_bootstrap.cc:492] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:48.218973 31779 raft_consensus.cc:729] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:48.219976 31609 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
W20260805 22:23:48.220702 32312 log.cc:924] Time spent T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.054s user 0.001s sys 0.000s
I20260805 22:23:48.221269 32306 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.224033 32306 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:48.219630 31779 consensus_queue.cc:260] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.227705 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
W20260805 22:23:48.228825 31667 leader_election.cc:341] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:48.230044 31779 ts_tablet_manager.cc:1428] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.056s user 0.007s sys 0.000s
I20260805 22:23:48.230999 31779 tablet_bootstrap.cc:492] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
W20260805 22:23:48.231346 31444 leader_election.cc:341] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:48.231743 31444 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:48.237758 32264 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:48.240826 31444 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:23:48.242187 31444 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:23:48.248484 32144 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.250869 32144 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:48.254274 32144 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:48.254714 32144 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:48.253710 32261 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:48.255266 32261 raft_consensus.cc:686] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:48.256093 32261 consensus_queue.cc:237] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.258062 32144 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:48.262456 32325 consensus_queue.cc:1035] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:48.269222 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
W20260805 22:23:48.277462 31444 leader_election.cc:341] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:48.277418 32171 consensus_queue.cc:1035] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.008s
I20260805 22:23:48.288221 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:48.289963 31601 raft_consensus.cc:3037] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:48.297055 31601 raft_consensus.cc:2454] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:48.298516 31768 tablet_bootstrap.cc:492] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:48.300132 32144 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.301337 31493 raft_consensus.cc:2379] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 1: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
W20260805 22:23:48.307107 31444 leader_election.cc:341] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:48.307554 31444 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:48.309569 31667 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:48.314014 31444 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:48.314323 31771 tablet_bootstrap.cc:654] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:48.310720 32264 raft_consensus.cc:2726] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:48.321841 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:48.312786 32131 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260805 22:23:48.370033 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 54279us
I20260805 22:23:48.379251 31768 tablet_bootstrap.cc:654] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:48.376771 32321 raft_consensus.cc:2781] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
W20260805 22:23:48.407341 31423 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
W20260805 22:23:48.404920 31444 leader_election.cc:341] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:48.418596 31779 tablet_bootstrap.cc:654] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:48.409960 32321 raft_consensus.cc:686] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:48.436661 32144 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:48.449724 32321 consensus_queue.cc:237] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.452427 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:48.452888 32144 consensus_queue.cc:1035] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:48.470535 31357 catalog_manager.cc:5495] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:48.476068 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:48.485056 31713 raft_consensus.cc:2379] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 1: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:23:48.493721 31771 tablet_bootstrap.cc:492] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:48.494197 31771 ts_tablet_manager.cc:1397] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.276s user 0.010s sys 0.005s
I20260805 22:23:48.499610 31779 tablet_bootstrap.cc:492] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:48.500072 31779 ts_tablet_manager.cc:1397] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.269s user 0.014s sys 0.001s
I20260805 22:23:48.497041 32133 consensus_queue.cc:1035] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20260805 22:23:48.503515 31771 raft_consensus.cc:348] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.504217 31771 raft_consensus.cc:374] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:48.504473 31771 raft_consensus.cc:729] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:48.505060 31771 consensus_queue.cc:260] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.507215 31771 ts_tablet_manager.cc:1428] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.013s user 0.003s sys 0.002s
I20260805 22:23:48.508219 31771 tablet_bootstrap.cc:492] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:48.502432 31779 raft_consensus.cc:348] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.509366 31779 raft_consensus.cc:374] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:48.509649 31779 raft_consensus.cc:729] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:48.510247 31779 consensus_queue.cc:260] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.516327 31768 tablet_bootstrap.cc:492] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:48.525447 31779 ts_tablet_manager.cc:1428] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.025s user 0.008s sys 0.000s
W20260805 22:23:48.525009 31444 leader_election.cc:341] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:48.526896 31768 ts_tablet_manager.cc:1397] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.229s user 0.014s sys 0.019s
I20260805 22:23:48.543473 31444 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:48.544152 31779 tablet_bootstrap.cc:492] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:48.529328 32323 consensus_queue.cc:1035] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.022s
I20260805 22:23:48.528754 32133 consensus_queue.cc:1035] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.024s
I20260805 22:23:48.556654 32323 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260805 22:23:48.564530 31492 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:48.568991 31357 catalog_manager.cc:5495] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:48.578287 31768 raft_consensus.cc:348] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.582134 31768 raft_consensus.cc:374] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:48.582464 31768 raft_consensus.cc:729] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:48.583058 31768 consensus_queue.cc:260] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.590442 32133 consensus_queue.cc:1035] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:48.591161 32323 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:23:48.593475 32345 log.cc:924] Time spent T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.050s user 0.000s sys 0.001s
I20260805 22:23:48.593647 32323 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.599462 31771 tablet_bootstrap.cc:654] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:48.644862 32302 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:48.645454 32302 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.646075 32337 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:48.646533 32337 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.648355 32337 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:48.644785 31779 tablet_bootstrap.cc:654] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:48.651568 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:48.658607 31768 ts_tablet_manager.cc:1428] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.131s user 0.005s sys 0.001s
I20260805 22:23:48.659601 31768 tablet_bootstrap.cc:492] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:48.662395 32131 consensus_queue.cc:1035] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:48.667657 32324 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:48.668187 32324 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.664924 32079 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:48.680471 32079 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.676523 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:48.676573 31714 raft_consensus.cc:2379] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 1: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:23:48.682464 32079 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:48.663902 32323 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:23:48.689697 31444 leader_election.cc:341] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:48.690491 31779 tablet_bootstrap.cc:492] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:48.691216 31779 ts_tablet_manager.cc:1397] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.163s user 0.008s sys 0.006s
I20260805 22:23:48.691737 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:48.711526 31768 tablet_bootstrap.cc:654] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:48.718160 31554 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:48.713913 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:48.722887 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:23:48.726996 32352 log.cc:924] Time spent T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.074s user 0.001s sys 0.000s
I20260805 22:23:48.744796 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:48.839681 31779 raft_consensus.cc:348] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.840345 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:48.715632 32302 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:23:48.842159 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 124094us
I20260805 22:23:48.844065 32337 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:48.851773 32337 raft_consensus.cc:2726] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:48.859521 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:48.860190 31602 raft_consensus.cc:2454] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:48.840481 31779 raft_consensus.cc:374] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:48.861004 31779 raft_consensus.cc:729] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:48.861722 31779 consensus_queue.cc:260] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.863348 31779 ts_tablet_manager.cc:1428] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.172s user 0.005s sys 0.014s
I20260805 22:23:48.864316 31779 tablet_bootstrap.cc:492] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:48.870695 31779 tablet_bootstrap.cc:654] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:48.871847 31667 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:48.873136 32131 raft_consensus.cc:2781] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:48.873582 32131 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:48.873973 32131 raft_consensus.cc:3037] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
W20260805 22:23:48.884501 31444 leader_election.cc:341] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:48.891709 31779 tablet_bootstrap.cc:492] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:48.905557 31779 ts_tablet_manager.cc:1397] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.041s user 0.015s sys 0.012s
W20260805 22:23:48.907490 31444 leader_election.cc:341] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:48.899047 31667 leader_election.cc:341] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:48.910420 31444 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:23:48.921566 31444 leader_election.cc:341] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:48.921988 31444 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:48.927709 32333 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:48.928367 32333 raft_consensus.cc:2726] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:48.931478 31768 tablet_bootstrap.cc:492] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:48.931954 31768 ts_tablet_manager.cc:1397] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.273s user 0.005s sys 0.009s
I20260805 22:23:48.934314 31768 raft_consensus.cc:348] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.934959 31768 raft_consensus.cc:374] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:48.935228 31768 raft_consensus.cc:729] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:48.935783 31768 consensus_queue.cc:260] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.880815 32131 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.939040 32131 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:48.940132 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:48.940712 31609 raft_consensus.cc:3037] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:48.896308 32324 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:48.908087 31779 raft_consensus.cc:348] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.945914 31779 raft_consensus.cc:374] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:48.946177 31779 raft_consensus.cc:729] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:48.946761 31779 consensus_queue.cc:260] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.951361 31609 raft_consensus.cc:2454] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:23:48.952306 31667 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:48.953159 32133 raft_consensus.cc:2781] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:48.953644 32133 raft_consensus.cc:686] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:23:48.954384 32133 consensus_queue.cc:237] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.954988 31771 tablet_bootstrap.cc:492] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:48.958051 31771 ts_tablet_manager.cc:1397] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.450s user 0.010s sys 0.008s
I20260805 22:23:48.961697 31771 raft_consensus.cc:348] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.966483 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:48.969990 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:48.971124 31779 ts_tablet_manager.cc:1428] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.065s user 0.005s sys 0.008s
I20260805 22:23:48.973515 31779 tablet_bootstrap.cc:492] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:48.974397 31768 ts_tablet_manager.cc:1428] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.042s user 0.001s sys 0.005s
I20260805 22:23:48.971449 32306 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:48.975389 31768 tablet_bootstrap.cc:492] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:48.975196 32306 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.977401 32306 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:48.982167 31771 raft_consensus.cc:374] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:48.982527 31771 raft_consensus.cc:729] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:48.982429 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:48.985612 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:48.984687 31771 consensus_queue.cc:260] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:48.988440 31493 raft_consensus.cc:3037] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:48.997218 31493 raft_consensus.cc:2454] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:23:48.999445 31771 ts_tablet_manager.cc:1428] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.040s user 0.004s sys 0.000s
I20260805 22:23:49.000744 31779 tablet_bootstrap.cc:654] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:49.000900 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:49.003531 31604 raft_consensus.cc:2379] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 1: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:23:49.002868 32328 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:49.004707 31771 tablet_bootstrap.cc:492] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:49.004328 32328 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.006430 32328 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:49.007790 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:49.008685 31714 raft_consensus.cc:3037] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:49.010568 32326 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:49.012684 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:49.013296 31492 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
W20260805 22:23:49.013891 31554 leader_election.cc:341] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:49.019971 31667 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:49.022598 31554 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:49.023838 32367 raft_consensus.cc:2726] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:49.024525 32328 raft_consensus.cc:2781] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:49.024880 32328 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:49.025184 32328 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:49.027603 31779 tablet_bootstrap.cc:492] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:49.028342 31779 ts_tablet_manager.cc:1397] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.055s user 0.009s sys 0.001s
I20260805 22:23:49.033836 31771 tablet_bootstrap.cc:654] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:49.044480 32328 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.047067 32328 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:49.048395 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:23:49.049580 31554 leader_election.cc:341] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:49.051313 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:49.051936 31492 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:49.058416 31779 raft_consensus.cc:348] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.059226 31779 raft_consensus.cc:374] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:49.059489 31779 raft_consensus.cc:729] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:49.058846 31359 catalog_manager.cc:5495] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:49.060101 31779 consensus_queue.cc:260] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.065181 31492 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:49.067374 31444 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:49.068264 32333 raft_consensus.cc:2726] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:23:49.070997 31554 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:49.073350 32340 raft_consensus.cc:2781] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:49.073650 32171 consensus_queue.cc:1035] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:49.075059 32340 raft_consensus.cc:686] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:49.073268 31779 ts_tablet_manager.cc:1428] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.045s user 0.005s sys 0.002s
I20260805 22:23:49.076190 31779 tablet_bootstrap.cc:492] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:49.075866 32340 consensus_queue.cc:237] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.086611 32171 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:49.086892 32326 consensus_queue.cc:1035] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:49.095127 32329 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:49.095644 32329 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.100244 32329 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:49.100355 31771 tablet_bootstrap.cc:492] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:49.100916 31771 ts_tablet_manager.cc:1397] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.096s user 0.024s sys 0.016s
I20260805 22:23:49.105261 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:49.117816 31708 raft_consensus.cc:2379] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 1: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:23:49.121330 31768 tablet_bootstrap.cc:654] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:49.122948 32133 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:49.123440 32133 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.127655 31771 raft_consensus.cc:348] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.128345 31771 raft_consensus.cc:374] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:49.128578 31771 raft_consensus.cc:729] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:49.118302 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:49.135372 31492 raft_consensus.cc:2379] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 1: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:23:49.135159 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:49.136101 31609 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:49.136096 32133 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:49.137193 31667 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:49.133544 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:49.138304 32133 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:49.138706 32133 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:49.138794 31554 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:49.139101 32133 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:49.140128 32328 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:49.141987 32367 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:49.142091 31771 consensus_queue.cc:260] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.145613 31502 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:49.150092 31779 tablet_bootstrap.cc:654] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:49.142498 32367 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.156513 32367 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:49.216279 31771 ts_tablet_manager.cc:1428] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.115s user 0.005s sys 0.004s
I20260805 22:23:49.229104 32328 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:49.238859 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:49.241533 31771 tablet_bootstrap.cc:492] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:49.239598 31502 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:49.224645 32133 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.314791 31667 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:23:49.315629 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 58382.9us
I20260805 22:23:49.322705 32373 raft_consensus.cc:2781] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:49.322862 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:49.323203 32373 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:49.323486 31501 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:49.323520 32373 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:49.333846 32133 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:49.336377 31771 tablet_bootstrap.cc:654] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:49.337630 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:49.353353 31501 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:23:49.355180 31667 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:49.355710 31779 tablet_bootstrap.cc:492] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:49.356127 31779 ts_tablet_manager.cc:1397] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.280s user 0.034s sys 0.052s
I20260805 22:23:49.358192 32367 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:49.359047 32368 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:49.362358 32368 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.364563 32368 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:49.359468 32367 raft_consensus.cc:686] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:23:49.366020 32367 consensus_queue.cc:237] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.374480 31779 raft_consensus.cc:348] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.375195 31779 raft_consensus.cc:374] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:49.375459 31779 raft_consensus.cc:729] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:49.376083 31779 consensus_queue.cc:260] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.376988 32373 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:49.382042 31667 leader_election.cc:341] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:49.387558 32373 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:49.388729 31771 tablet_bootstrap.cc:492] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:49.389187 31771 ts_tablet_manager.cc:1397] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.157s user 0.010s sys 0.009s
I20260805 22:23:49.390019 31768 tablet_bootstrap.cc:492] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:49.376785 31357 catalog_manager.cc:5495] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:49.378279 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:49.390499 31768 ts_tablet_manager.cc:1397] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.415s user 0.012s sys 0.030s
I20260805 22:23:49.397241 31358 catalog_manager.cc:5495] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:49.400859 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:49.400547 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:49.379863 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:49.390667 31609 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:49.378211 31779 ts_tablet_manager.cc:1428] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.022s user 0.002s sys 0.005s
I20260805 22:23:49.391516 31771 raft_consensus.cc:348] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.378917 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:49.401520 31493 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:49.402343 31602 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:49.402742 31779 tablet_bootstrap.cc:492] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:49.403828 31667 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:49.404898 32373 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:49.405277 32373 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:49.405818 32373 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:49.408675 31493 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
W20260805 22:23:49.410809 31667 leader_election.cc:341] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:49.411721 31667 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:49.412376 32367 raft_consensus.cc:2781] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:49.412891 32367 raft_consensus.cc:686] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:23:49.413357 31779 tablet_bootstrap.cc:654] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:49.413827 32367 consensus_queue.cc:237] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:49.416230 31667 leader_election.cc:341] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:49.417585 31768 raft_consensus.cc:348] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.418999 31768 raft_consensus.cc:374] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:49.419261 31768 raft_consensus.cc:729] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:49.420023 31768 consensus_queue.cc:260] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.422766 31768 ts_tablet_manager.cc:1428] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260805 22:23:49.423530 31771 raft_consensus.cc:374] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:49.423925 31771 raft_consensus.cc:729] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:49.425750 31771 consensus_queue.cc:260] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.423079 32373 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.425886 31609 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:23:49.429884 31768 tablet_bootstrap.cc:492] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:49.430883 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:49.442011 32373 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:23:49.450615 31667 leader_election.cc:341] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:49.455628 31771 ts_tablet_manager.cc:1428] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.066s user 0.004s sys 0.001s
I20260805 22:23:49.456712 31771 tablet_bootstrap.cc:492] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:49.464417 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:49.467546 31609 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:49.481370 32373 consensus_queue.cc:1035] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:49.496351 31768 tablet_bootstrap.cc:654] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:49.501565 31771 tablet_bootstrap.cc:654] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:49.518229 31779 tablet_bootstrap.cc:492] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:49.518764 31779 ts_tablet_manager.cc:1397] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.116s user 0.029s sys 0.037s
I20260805 22:23:49.521709 31768 tablet_bootstrap.cc:492] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:49.528846 31768 ts_tablet_manager.cc:1397] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.099s user 0.013s sys 0.000s
I20260805 22:23:49.533358 31779 raft_consensus.cc:348] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.534453 31779 raft_consensus.cc:374] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:49.535179 31779 raft_consensus.cc:729] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:49.536425 31779 consensus_queue.cc:260] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.537920 31771 tablet_bootstrap.cc:492] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:49.538378 31771 ts_tablet_manager.cc:1397] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.082s user 0.011s sys 0.000s
I20260805 22:23:49.539143 31779 ts_tablet_manager.cc:1428] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20260805 22:23:49.540863 31779 tablet_bootstrap.cc:492] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
W20260805 22:23:49.540999 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 202518us
I20260805 22:23:49.545833 31609 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:23:49.542492 31358 catalog_manager.cc:5495] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:49.546770 31667 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:23:49.560869 31768 raft_consensus.cc:348] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.561719 31768 raft_consensus.cc:374] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:49.562021 31768 raft_consensus.cc:729] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:49.562641 31768 consensus_queue.cc:260] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.560376 31779 tablet_bootstrap.cc:654] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:49.553629 32373 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:49.573519 32373 raft_consensus.cc:686] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:23:49.602370 31771 raft_consensus.cc:348] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.603241 31771 raft_consensus.cc:374] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:49.603529 31771 raft_consensus.cc:729] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:49.604139 31771 consensus_queue.cc:260] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.611660 32306 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:49.612213 32306 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.614318 32306 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:49.615727 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:49.616374 31609 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:49.617604 31667 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:49.618391 32368 consensus_queue.cc:1035] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:49.619683 32368 raft_consensus.cc:2781] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:49.619990 32368 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:49.620289 32368 raft_consensus.cc:3037] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:49.622267 32373 consensus_queue.cc:237] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.631294 31771 ts_tablet_manager.cc:1428] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.093s user 0.008s sys 0.000s
I20260805 22:23:49.645073 31779 tablet_bootstrap.cc:492] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:49.645710 31779 ts_tablet_manager.cc:1397] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.105s user 0.010s sys 0.003s
I20260805 22:23:49.666258 31771 tablet_bootstrap.cc:492] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:49.674000 31779 raft_consensus.cc:348] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.680640 31779 raft_consensus.cc:374] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:49.680960 31779 raft_consensus.cc:729] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:49.682042 31779 consensus_queue.cc:260] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.685963 32403 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:49.686367 31779 ts_tablet_manager.cc:1428] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.040s user 0.007s sys 0.001s
I20260805 22:23:49.687829 31779 tablet_bootstrap.cc:492] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:49.686517 32403 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.700685 32368 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:23:49.721781 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 78672.7us
W20260805 22:23:49.722589 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 61834us
I20260805 22:23:49.731410 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:49.732138 31492 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:49.740113 32368 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:23:49.741567 31423 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260805 22:23:49.740854 32403 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:49.751029 31779 tablet_bootstrap.cc:654] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:49.754379 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:49.754952 31609 raft_consensus.cc:3037] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:49.758603 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:49.759208 31492 raft_consensus.cc:3037] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:49.759681 31771 tablet_bootstrap.cc:654] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:49.762351 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:49.763780 31358 catalog_manager.cc:5495] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:49.735040 32389 consensus_queue.cc:1035] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.038s
I20260805 22:23:49.774819 31768 ts_tablet_manager.cc:1428] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.244s user 0.023s sys 0.015s
I20260805 22:23:49.775839 31768 tablet_bootstrap.cc:492] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:49.776654 32368 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:49.777172 32368 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.789366 32368 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:49.789736 31492 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:23:49.794166 31609 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
W20260805 22:23:49.846854 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 52644.7us
I20260805 22:23:49.908186 31768 tablet_bootstrap.cc:654] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:49.915084 31667 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:23:49.913899 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 53177us
I20260805 22:23:49.909200 32427 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:23:49.922928 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 63920.1us
W20260805 22:23:49.925154 31444 leader_election.cc:341] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:49.923790 32427 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:49.932061 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:49.944679 31603 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:49.927430 32373 raft_consensus.cc:2781] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election won for term 1
W20260805 22:23:49.998273 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 74144.3us
I20260805 22:23:50.041605 32427 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:49.978037 32373 raft_consensus.cc:686] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
W20260805 22:23:50.043401 32419 log.cc:924] Time spent T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260805 22:23:50.054435 32368 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:23:50.078154 31554 consensus_peers.cc:487] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c -> Peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Couldn't send request to peer feb5e7ab4b1541beaf2674c3c5500385. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:50.080869 31779 tablet_bootstrap.cc:492] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:50.083300 31779 ts_tablet_manager.cc:1397] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.396s user 0.015s sys 0.030s
I20260805 22:23:50.096146 31667 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:50.072089 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:50.073431 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:50.074717 32368 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.108413 32321 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:50.112329 32321 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.102669 32367 raft_consensus.cc:2781] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:50.114260 32321 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:50.114461 32367 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:50.114883 32367 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:50.105779 31502 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:50.117369 32131 consensus_queue.cc:1035] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.013s
I20260805 22:23:50.076913 32373 consensus_queue.cc:237] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.122541 31708 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:50.147558 31771 tablet_bootstrap.cc:492] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:50.148013 31771 ts_tablet_manager.cc:1397] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.482s user 0.015s sys 0.018s
I20260805 22:23:50.152634 32367 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.154600 32367 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:50.119328 32368 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:50.165598 31771 raft_consensus.cc:348] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.166417 31771 raft_consensus.cc:374] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:50.166666 31771 raft_consensus.cc:729] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:50.167263 31771 consensus_queue.cc:260] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.177418 31771 ts_tablet_manager.cc:1428] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.029s user 0.007s sys 0.000s
I20260805 22:23:50.178686 31771 tablet_bootstrap.cc:492] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994: Bootstrap starting.
I20260805 22:23:50.189363 31779 raft_consensus.cc:348] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.197777 31779 raft_consensus.cc:374] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:50.198153 31779 raft_consensus.cc:729] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:50.195547 31771 tablet_bootstrap.cc:654] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994: Neither blocks nor log segments found. Creating new log.
W20260805 22:23:50.207826 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 116678us
I20260805 22:23:50.199589 31768 tablet_bootstrap.cc:492] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:50.208493 31768 ts_tablet_manager.cc:1397] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.433s user 0.013s sys 0.008s
I20260805 22:23:50.212884 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:50.213629 31502 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:50.225464 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:50.226210 31502 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 0.
I20260805 22:23:50.236841 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:50.237722 31602 raft_consensus.cc:2379] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 1: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:23:50.205479 31779 consensus_queue.cc:260] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.265231 31779 ts_tablet_manager.cc:1428] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.182s user 0.006s sys 0.000s
I20260805 22:23:50.268074 31358 catalog_manager.cc:5495] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:50.269042 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:50.271271 31444 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:23:50.271407 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:50.272040 32317 raft_consensus.cc:2781] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:50.272415 32317 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:50.272745 32317 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:50.277304 31768 raft_consensus.cc:348] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.277990 31768 raft_consensus.cc:374] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:50.278251 31768 raft_consensus.cc:729] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:50.272148 31713 raft_consensus.cc:2379] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 1: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:23:50.280046 31768 consensus_queue.cc:260] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.282367 31768 ts_tablet_manager.cc:1428] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.074s user 0.006s sys 0.000s
I20260805 22:23:50.284327 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:50.284912 31501 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:50.289484 32437 consensus_queue.cc:1035] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:50.290731 31667 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:50.291497 32373 raft_consensus.cc:2781] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:50.291827 32373 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:50.292131 32373 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:50.293154 31501 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:23:50.296113 32317 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.298270 32317 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:50.299288 32373 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.302253 31604 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:50.305850 31501 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:50.301251 32373 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:50.310842 31667 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:50.326812 32373 raft_consensus.cc:2781] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:50.327296 32373 raft_consensus.cc:686] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:23:50.328150 32373 consensus_queue.cc:237] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.332005 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:50.332783 31707 raft_consensus.cc:2379] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 1: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:23:50.314090 31768 tablet_bootstrap.cc:492] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:50.314508 31554 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:50.335175 32426 raft_consensus.cc:2781] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:50.335520 32426 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:50.335829 32426 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:50.338711 31771 tablet_bootstrap.cc:492] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994: No bootstrap required, opened a new log
I20260805 22:23:50.339236 31771 ts_tablet_manager.cc:1397] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994: Time spent bootstrapping tablet: real 0.161s user 0.011s sys 0.000s
I20260805 22:23:50.340572 32439 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:50.341055 32439 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.344059 32439 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:50.346939 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:50.348287 31771 raft_consensus.cc:348] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.317531 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:50.356036 31771 raft_consensus.cc:374] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:50.358321 31771 raft_consensus.cc:729] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Initialized, Role: FOLLOWER
I20260805 22:23:50.359369 31771 consensus_queue.cc:260] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.349226 31444 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:50.354254 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:50.359987 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:50.358592 31707 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:50.360662 31501 raft_consensus.cc:2379] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 1: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:23:50.364580 32437 consensus_queue.cc:1035] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:50.316659 31779 tablet_bootstrap.cc:492] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:50.366127 32364 raft_consensus.cc:2726] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:50.369900 32426 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.371170 31603 raft_consensus.cc:2379] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 1: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:23:50.371960 32426 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:50.377033 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:50.378314 31768 tablet_bootstrap.cc:654] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:50.381860 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:50.383083 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:50.383754 31608 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:23:50.388465 31771 ts_tablet_manager.cc:1428] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994: Time spent starting tablet: real 0.049s user 0.005s sys 0.000s
I20260805 22:23:50.380386 31501 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:23:50.404968 31779 tablet_bootstrap.cc:654] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
W20260805 22:23:50.405889 31667 leader_election.cc:341] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
W20260805 22:23:50.407181 31667 leader_election.cc:341] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:50.407686 31667 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:50.420200 32433 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:23:50.420681 32433 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:50.421021 32433 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:50.423563 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:50.424495 31492 raft_consensus.cc:2379] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 1: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
W20260805 22:23:50.425028 31444 leader_election.cc:341] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:50.425464 31444 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:50.425805 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:50.426709 32410 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260805 22:23:50.405671 32306 consensus_queue.cc:1035] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s
I20260805 22:23:50.429682 32437 consensus_queue.cc:1035] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.021s
I20260805 22:23:50.416564 32302 consensus_queue.cc:1035] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.032s
W20260805 22:23:50.504959 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 96643us
I20260805 22:23:50.506742 31667 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:50.511690 31768 tablet_bootstrap.cc:492] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:50.516325 31768 ts_tablet_manager.cc:1397] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.202s user 0.011s sys 0.023s
I20260805 22:23:50.519018 32457 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260805 22:23:50.535324 32429 log.cc:924] Time spent T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.107s user 0.001s sys 0.001s
I20260805 22:23:50.534979 31768 raft_consensus.cc:348] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.536108 31768 raft_consensus.cc:374] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:50.536733 31768 raft_consensus.cc:729] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:50.537492 31768 consensus_queue.cc:260] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.553992 32328 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:50.554520 32328 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.556347 32328 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:50.542280 32433 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.562881 32433 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:50.564746 31768 ts_tablet_manager.cc:1428] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.047s user 0.007s sys 0.000s
I20260805 22:23:50.565805 31768 tablet_bootstrap.cc:492] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:50.572522 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:50.573166 31501 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:50.573791 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:50.577216 31603 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:50.579510 31768 tablet_bootstrap.cc:654] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:50.584079 31603 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:23:50.585062 31667 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:50.585894 32131 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:23:50.586335 32131 raft_consensus.cc:686] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:23:50.587164 32131 consensus_queue.cc:237] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.613296 31501 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:23:50.663303 32323 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:50.663864 32323 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.665863 32323 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:50.701243 31358 catalog_manager.cc:5495] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 0 to 1, leader changed from <none> to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:50.719118 32368 consensus_queue.cc:1035] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:50.725673 32373 consensus_queue.cc:1035] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260805 22:23:50.763440 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 146976us
I20260805 22:23:50.778702 31779 tablet_bootstrap.cc:492] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:50.779222 31779 ts_tablet_manager.cc:1397] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.463s user 0.017s sys 0.001s
I20260805 22:23:50.793962 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:50.796111 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:23:50.804839 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 372801us
W20260805 22:23:50.810040 31423 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260805 22:23:50.828920 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:50.843598 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:50.850085 31359 catalog_manager.cc:5495] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 0 to 2, leader changed from <none> to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:23:50.860158 31554 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:50.861172 31779 raft_consensus.cc:348] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.865878 31779 raft_consensus.cc:374] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:50.866187 31779 raft_consensus.cc:729] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:50.876154 32427 raft_consensus.cc:2726] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260805 22:23:50.878834 31779 consensus_queue.cc:260] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.883773 32406 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:50.884339 32406 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.890316 31768 tablet_bootstrap.cc:492] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:50.890734 31768 ts_tablet_manager.cc:1397] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.325s user 0.056s sys 0.094s
I20260805 22:23:50.894234 32406 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:23:50.938561 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 55124.2us
I20260805 22:23:50.861657 32131 consensus_queue.cc:1035] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.076s
I20260805 22:23:50.946997 31779 ts_tablet_manager.cc:1428] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.167s user 0.007s sys 0.000s
W20260805 22:23:50.948190 31444 leader_election.cc:341] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:50.958439 31779 tablet_bootstrap.cc:492] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385: Bootstrap starting.
I20260805 22:23:50.972801 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:50.974273 31609 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:50.989593 31768 raft_consensus.cc:348] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.990288 32131 consensus_queue.cc:1035] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260805 22:23:50.991254 31444 leader_election.cc:341] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Illegal state: must be running to vote when last-logged opid is not known
I20260805 22:23:50.992461 31444 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:50.990687 31768 raft_consensus.cc:374] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:50.993407 31768 raft_consensus.cc:729] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
W20260805 22:23:50.994477 32474 log.cc:924] Time spent T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.100s user 0.000s sys 0.003s
I20260805 22:23:50.995121 32425 raft_consensus.cc:2726] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:50.994146 31768 consensus_queue.cc:260] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:50.979539 31554 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:51.002104 32328 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:23:51.012265 31768 ts_tablet_manager.cc:1428] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.121s user 0.008s sys 0.000s
I20260805 22:23:51.014662 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:51.015377 31707 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 0.
I20260805 22:23:51.019889 31768 tablet_bootstrap.cc:492] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c: Bootstrap starting.
I20260805 22:23:51.030709 31444 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:51.034531 32171 raft_consensus.cc:2781] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:51.041915 32171 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:51.042279 32171 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:51.080271 31779 tablet_bootstrap.cc:654] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:51.082904 31594 raft_consensus.cc:1261] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260805 22:23:51.084291 32171 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:51.088395 32171 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:51.095528 32457 consensus_queue.cc:1035] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:51.097589 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:51.098306 31709 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:51.108340 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:51.109004 31596 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:51.141554 31492 raft_consensus.cc:1261] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260805 22:23:51.147339 32302 consensus_queue.cc:1035] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:51.155648 31709 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:51.156517 31768 tablet_bootstrap.cc:654] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c: Neither blocks nor log segments found. Creating new log.
I20260805 22:23:51.156872 31444 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:51.160038 32171 raft_consensus.cc:2781] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:51.160557 32171 raft_consensus.cc:686] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:51.161443 32171 consensus_queue.cc:237] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:51.165242 31596 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:51.176388 31768 tablet_bootstrap.cc:492] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c: No bootstrap required, opened a new log
I20260805 22:23:51.176848 31768 ts_tablet_manager.cc:1397] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c: Time spent bootstrapping tablet: real 0.157s user 0.009s sys 0.006s
I20260805 22:23:51.179280 31768 raft_consensus.cc:348] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:51.179951 31768 raft_consensus.cc:374] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:51.180192 31768 raft_consensus.cc:729] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Initialized, Role: FOLLOWER
I20260805 22:23:51.180766 31768 consensus_queue.cc:260] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:51.182909 31768 ts_tablet_manager.cc:1428] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260805 22:23:51.197045 31357 catalog_manager.cc:5495] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 1, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:51.223495 31779 tablet_bootstrap.cc:492] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385: No bootstrap required, opened a new log
I20260805 22:23:51.223951 31779 ts_tablet_manager.cc:1397] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385: Time spent bootstrapping tablet: real 0.270s user 0.020s sys 0.033s
I20260805 22:23:51.226606 31779 raft_consensus.cc:348] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:51.227286 31779 raft_consensus.cc:374] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260805 22:23:51.227552 31779 raft_consensus.cc:729] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Initialized, Role: FOLLOWER
I20260805 22:23:51.228349 31779 consensus_queue.cc:260] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:51.230809 31779 ts_tablet_manager.cc:1428] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260805 22:23:51.231693 32426 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:51.232152 32426 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:51.235915 32426 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:51.237105 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:51.237970 31709 raft_consensus.cc:2379] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 1: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:23:51.239054 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:51.239917 31492 raft_consensus.cc:2379] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 1: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:23:51.241034 31554 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:51.242390 32477 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:51.262651 32477 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260805 22:23:51.318141 32477 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:51.318642 32477 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:51.331586 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:51.332334 31492 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:51.334160 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:51.332463 32477 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:51.334898 31706 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 0.
I20260805 22:23:51.336292 31554 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:51.338845 32477 raft_consensus.cc:2781] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260805 22:23:51.339250 32477 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:51.339603 32477 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:51.367652 32477 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:51.370492 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:51.371165 31492 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:51.378582 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:51.379336 31706 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 0 FOLLOWER]: Advancing to term 1
I20260805 22:23:51.382920 31492 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:51.383857 32477 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:51.384536 31554 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:51.385680 32477 raft_consensus.cc:2781] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader election won for term 1
I20260805 22:23:51.386715 31706 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:23:51.386914 32477 raft_consensus.cc:686] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:23:51.387950 32477 consensus_queue.cc:237] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:51.399657 31357 catalog_manager.cc:5495] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 0 to 1, leader changed from <none> to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:51.620450 32171 consensus_queue.cc:1035] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:51.641997 32322 consensus_queue.cc:1035] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260805 22:23:51.852769 31423 auto_rebalancer.cc:227] Could not retrieve cluster info: Service unavailable: Tablet not running
I20260805 22:23:51.955754 32329 consensus_queue.cc:1035] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:51.974108 32329 consensus_queue.cc:1035] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:52.313052 32306 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:52.313522 32306 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:52.315073 32306 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:52.315728 32171 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:23:52.315979 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:23:52.316133 32171 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:52.317009 31494 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:23:52.317353 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:52.317901 32171 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:52.317955 31604 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:23:52.318493 31667 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:23:52.319320 32306 raft_consensus.cc:2781] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:23:52.319662 32306 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:52.320170 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:23:52.320195 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:23:52.320367 32306 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:52.320811 31604 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:23:52.321758 31444 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:23:52.322650 32171 raft_consensus.cc:2781] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:23:52.322996 32171 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:23:52.323436 32171 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:52.328680 32171 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:52.330278 32171 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:52.330606 32306 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:52.331224 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:52.332331 31604 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:23:52.332607 31709 raft_consensus.cc:2379] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 2: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:23:52.333355 32306 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:52.334802 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:23:52.331645 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:23:52.335429 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:23:52.336052 31706 raft_consensus.cc:2379] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 2: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:23:52.335498 31494 raft_consensus.cc:2379] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 2: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:23:52.337845 31667 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:23:52.338675 32306 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:23:52.339937 31604 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:23:52.340811 31444 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:23:52.341511 32171 raft_consensus.cc:2781] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:23:52.341943 32171 raft_consensus.cc:686] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 2 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:23:52.342615 32171 consensus_queue.cc:237] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:23:52.364982 31357 catalog_manager.cc:5495] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 0 to 2, leader changed from <none> to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 2 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:52.781781 31709 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260805 22:23:52.782783 32322 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260805 22:23:52.799463 31596 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260805 22:23:52.801661 32322 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260805 22:23:53.586282 29783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260805 22:23:53.612186 32516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:23:53.612241 32514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260805 22:23:53.614008 32518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260805 22:23:53.617426 29783 server_base.cc:1034] running on GCE node
I20260805 22:23:53.618887 29783 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260805 22:23:53.619104 29783 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260805 22:23:53.619278 29783 hybrid_clock.cc:648] HybridClock initialized: now 1785968633619264 us; error 0 us; skew 500 ppm
I20260805 22:23:53.619896 29783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260805 22:23:53.630010 29783 webserver.cc:460] Webserver started at http://127.29.21.196:45837/ using document root <none> and password file <none>
I20260805 22:23:53.630627 29783 fs_manager.cc:359] Metadata directory not provided
I20260805 22:23:53.630849 29783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260805 22:23:53.631173 29783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260805 22:23:53.632467 29783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-3-root/instance:
uuid: "6bf8c694a0ba40978195ae044a917caf"
format_stamp: "Formatted at 2026-08-05 22:23:53 on dist-test-slave-f0tn"
I20260805 22:23:53.637529 29783 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260805 22:23:53.641060 32528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:53.641911 29783 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260805 22:23:53.642187 29783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-3-root
uuid: "6bf8c694a0ba40978195ae044a917caf"
format_stamp: "Formatted at 2026-08-05 22:23:53 on dist-test-slave-f0tn"
I20260805 22:23:53.642457 29783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-3-root
metadata directory: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-3-root
1 data directories: /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-3-root/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260805 22:23:53.684257 29783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260805 22:23:53.685983 29783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260805 22:23:53.688369 29783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260805 22:23:53.691355 29783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260805 22:23:53.691627 29783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:53.691923 29783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260805 22:23:53.692135 29783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260805 22:23:53.806535 29783 rpc_server.cc:307] RPC server started. Bound to: 127.29.21.196:41801
I20260805 22:23:53.807619 32633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.21.196:41801 every 8 connection(s)
I20260805 22:23:53.812232 29783 auto_leader_rebalancer-test.cc:413] add a tserver: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:23:53.837769 32634 heartbeater.cc:347] Connected to a master server at 127.29.21.254:43401
I20260805 22:23:53.838320 32634 heartbeater.cc:464] Registering TS with master...
I20260805 22:23:53.839211 32634 heartbeater.cc:511] Master 127.29.21.254:43401 requested a full tablet report, sending...
I20260805 22:23:53.841468 31357 ts_manager.cc:194] Registered new tserver with Master: 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:23:53.843747 31357 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.0.1:47946
I20260805 22:23:54.004208 31704 consensus_queue.cc:237] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:54.011941 31604 raft_consensus.cc:1261] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:23:54.014631 32306 consensus_queue.cc:1035] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:54.030197 31492 raft_consensus.cc:1261] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:23:54.034339 32306 consensus_queue.cc:1035] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:54.054746 31494 raft_consensus.cc:2932] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:54.048278 32306 raft_consensus.cc:2932] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:54.062666 31604 raft_consensus.cc:2932] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:54.077215 31359 catalog_manager.cc:5495] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260805 22:23:54.128846 31668 consensus_peers.cc:487] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 58a08ace2d684c14972d495f4a387a85. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:54.141880 31604 consensus_queue.cc:237] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:54.198472 31706 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:23:54.200664 32639 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:54.205068 31492 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:23:54.206820 32639 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260805 22:23:54.237537 31556 consensus_peers.cc:487] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 39e7b32cb2f646828a02f900ec2d47f1. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:54.244225 32477 raft_consensus.cc:2932] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:54.247364 31492 raft_consensus.cc:2932] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:54.267520 31357 catalog_manager.cc:5495] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:54.274655 31604 consensus_queue.cc:237] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:54.307487 31704 raft_consensus.cc:2932] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:54.320462 31706 raft_consensus.cc:1261] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:23:54.331727 32575 consensus_queue.cc:1035] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260805 22:23:54.335290 31556 consensus_peers.cc:487] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78cb483d17014b3281316012dbb6f6e1. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:54.347041 32639 raft_consensus.cc:2932] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:54.348660 31706 raft_consensus.cc:2932] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:54.353214 31502 raft_consensus.cc:1261] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:23:54.362512 32500 consensus_queue.cc:1035] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:54.368655 31356 catalog_manager.cc:5495] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:54.388000 31492 raft_consensus.cc:2932] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:54.416308 31492 consensus_queue.cc:237] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:54.423697 31706 raft_consensus.cc:1261] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:23:54.423803 31604 raft_consensus.cc:1261] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:23:54.425155 32322 consensus_queue.cc:1035] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:54.427048 32513 consensus_queue.cc:1035] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:54.437098 32513 raft_consensus.cc:2932] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:54.440133 31704 raft_consensus.cc:2932] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:54.444048 31604 raft_consensus.cc:2932] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:54.467347 31357 catalog_manager.cc:5495] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
W20260805 22:23:54.522128 31447 consensus_peers.cc:487] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe370453251f4300a8d90c397021595c. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:54.584435 32675 ts_tablet_manager.cc:927] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:23:54.592844 32675 tablet_copy_client.cc:316] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:44831
I20260805 22:23:54.627059 31729 tablet_copy_service.cc:140] P feb5e7ab4b1541beaf2674c3c5500385: Received BeginTabletCopySession request for tablet 58a08ace2d684c14972d495f4a387a85 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:58348)
I20260805 22:23:54.627722 31729 tablet_copy_service.cc:161] P feb5e7ab4b1541beaf2674c3c5500385: Beginning new tablet copy session on tablet 58a08ace2d684c14972d495f4a387a85 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:58348: session id = 6bf8c694a0ba40978195ae044a917caf-58a08ace2d684c14972d495f4a387a85
I20260805 22:23:54.671126 31729 tablet_copy_source_session.cc:205] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:23:54.677438 32675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 58a08ace2d684c14972d495f4a387a85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:54.695986 32675 tablet_copy_client.cc:791] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:23:54.696755 32675 tablet_copy_client.cc:655] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:23:54.702204 32675 tablet_copy_client.cc:531] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:23:54.715656 32675 tablet_bootstrap.cc:492] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:23:54.744750 32675 tablet_bootstrap.cc:492] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:23:54.747601 32675 tablet_bootstrap.cc:492] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:23:54.748384 32675 ts_tablet_manager.cc:1397] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.005s
I20260805 22:23:54.751073 32675 raft_consensus.cc:348] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:54.752004 32675 raft_consensus.cc:729] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:23:54.752676 32675 consensus_queue.cc:260] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:54.757157 32634 heartbeater.cc:503] Master 127.29.21.254:43401 was elected leader, sending a full tablet report...
I20260805 22:23:54.758667 32675 ts_tablet_manager.cc:1428] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.010s user 0.005s sys 0.003s
I20260805 22:23:54.761979 31729 tablet_copy_service.cc:342] P feb5e7ab4b1541beaf2674c3c5500385: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-58a08ace2d684c14972d495f4a387a85 received from {username='slave'} at 127.0.0.1:58348
I20260805 22:23:54.762425 31729 tablet_copy_service.cc:433] P feb5e7ab4b1541beaf2674c3c5500385: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-58a08ace2d684c14972d495f4a387a85 on tablet 58a08ace2d684c14972d495f4a387a85 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:23:54.791049 32675 ts_tablet_manager.cc:927] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:54.795094 32675 tablet_copy_client.cc:316] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:41897
I20260805 22:23:54.815138 31623 tablet_copy_service.cc:140] P 516cb07ce3064ccbbca35e912d62816c: Received BeginTabletCopySession request for tablet 39e7b32cb2f646828a02f900ec2d47f1 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:56384)
I20260805 22:23:54.815734 31623 tablet_copy_service.cc:161] P 516cb07ce3064ccbbca35e912d62816c: Beginning new tablet copy session on tablet 39e7b32cb2f646828a02f900ec2d47f1 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:56384: session id = 6bf8c694a0ba40978195ae044a917caf-39e7b32cb2f646828a02f900ec2d47f1
I20260805 22:23:54.820914 32689 ts_tablet_manager.cc:927] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:23:54.827210 31623 tablet_copy_source_session.cc:205] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:23:54.833431 32689 tablet_copy_client.cc:316] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:45607
I20260805 22:23:54.834182 32675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39e7b32cb2f646828a02f900ec2d47f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:54.851786 32675 tablet_copy_client.cc:791] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:23:54.852500 32675 tablet_copy_client.cc:655] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:23:54.852773 31516 tablet_copy_service.cc:140] P f8e9f4c756da4203917c464e2b179994: Received BeginTabletCopySession request for tablet fe370453251f4300a8d90c397021595c from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:52098)
I20260805 22:23:54.853363 31516 tablet_copy_service.cc:161] P f8e9f4c756da4203917c464e2b179994: Beginning new tablet copy session on tablet fe370453251f4300a8d90c397021595c from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:52098: session id = 6bf8c694a0ba40978195ae044a917caf-fe370453251f4300a8d90c397021595c
I20260805 22:23:54.861063 31516 tablet_copy_source_session.cc:205] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:23:54.864363 32689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe370453251f4300a8d90c397021595c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:54.871397 32675 tablet_copy_client.cc:531] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:23:54.898574 32675 tablet_bootstrap.cc:492] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:23:54.900467 32689 tablet_copy_client.cc:791] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:23:54.901278 32689 tablet_copy_client.cc:655] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:23:54.910662 32689 tablet_copy_client.cc:531] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:23:54.937407 32691 ts_tablet_manager.cc:927] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:54.941709 32689 tablet_bootstrap.cc:492] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:23:54.965059 32691 tablet_copy_client.cc:316] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:41897
I20260805 22:23:54.972177 31623 tablet_copy_service.cc:140] P 516cb07ce3064ccbbca35e912d62816c: Received BeginTabletCopySession request for tablet 78cb483d17014b3281316012dbb6f6e1 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:56384)
I20260805 22:23:54.972743 31623 tablet_copy_service.cc:161] P 516cb07ce3064ccbbca35e912d62816c: Beginning new tablet copy session on tablet 78cb483d17014b3281316012dbb6f6e1 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:56384: session id = 6bf8c694a0ba40978195ae044a917caf-78cb483d17014b3281316012dbb6f6e1
I20260805 22:23:54.973445 32675 tablet_bootstrap.cc:492] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:23:54.975962 32675 tablet_bootstrap.cc:492] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:23:54.976714 32675 ts_tablet_manager.cc:1397] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.078s user 0.021s sys 0.005s
I20260805 22:23:54.981343 31623 tablet_copy_source_session.cc:205] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:23:54.984622 32691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 78cb483d17014b3281316012dbb6f6e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:54.986497 32675 raft_consensus.cc:348] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:54.987419 32675 raft_consensus.cc:729] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:23:54.988073 32675 consensus_queue.cc:260] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:54.990398 32675 ts_tablet_manager.cc:1428] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260805 22:23:54.993548 31623 tablet_copy_service.cc:342] P 516cb07ce3064ccbbca35e912d62816c: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-39e7b32cb2f646828a02f900ec2d47f1 received from {username='slave'} at 127.0.0.1:56384
I20260805 22:23:54.993957 31623 tablet_copy_service.cc:433] P 516cb07ce3064ccbbca35e912d62816c: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-39e7b32cb2f646828a02f900ec2d47f1 on tablet 39e7b32cb2f646828a02f900ec2d47f1 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:23:55.002357 32689 tablet_bootstrap.cc:492] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:23:55.003350 32689 tablet_bootstrap.cc:492] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:23:55.003468 32691 tablet_copy_client.cc:791] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:23:55.004025 32691 tablet_copy_client.cc:655] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:23:55.004103 32689 ts_tablet_manager.cc:1397] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.063s user 0.026s sys 0.000s
I20260805 22:23:55.009196 32689 raft_consensus.cc:348] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:55.010215 32689 raft_consensus.cc:729] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:23:55.010907 32691 tablet_copy_client.cc:531] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:23:55.011477 32689 consensus_queue.cc:260] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:55.014613 32689 ts_tablet_manager.cc:1428] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20260805 22:23:55.017475 31516 tablet_copy_service.cc:342] P f8e9f4c756da4203917c464e2b179994: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-fe370453251f4300a8d90c397021595c received from {username='slave'} at 127.0.0.1:52098
I20260805 22:23:55.017867 31516 tablet_copy_service.cc:433] P f8e9f4c756da4203917c464e2b179994: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-fe370453251f4300a8d90c397021595c on tablet fe370453251f4300a8d90c397021595c with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:23:55.022861 32691 tablet_bootstrap.cc:492] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:23:55.054056 32691 tablet_bootstrap.cc:492] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:23:55.054987 32691 tablet_bootstrap.cc:492] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:23:55.055732 32691 ts_tablet_manager.cc:1397] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.033s user 0.022s sys 0.011s
I20260805 22:23:55.058430 32691 raft_consensus.cc:348] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:55.059363 32691 raft_consensus.cc:729] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:23:55.060024 32691 consensus_queue.cc:260] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:55.066761 32691 ts_tablet_manager.cc:1428] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260805 22:23:55.069505 31623 tablet_copy_service.cc:342] P 516cb07ce3064ccbbca35e912d62816c: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-78cb483d17014b3281316012dbb6f6e1 received from {username='slave'} at 127.0.0.1:56384
I20260805 22:23:55.069913 31623 tablet_copy_service.cc:433] P 516cb07ce3064ccbbca35e912d62816c: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-78cb483d17014b3281316012dbb6f6e1 on tablet 78cb483d17014b3281316012dbb6f6e1 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:23:55.094725 32594 raft_consensus.cc:1203] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:23:55.330737 32594 raft_consensus.cc:1203] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:23:55.374469 32594 raft_consensus.cc:1203] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:23:55.522837 32594 raft_consensus.cc:1203] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:23:55.639634 32649 raft_consensus.cc:1050] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:23:55.642361 32649 consensus_queue.cc:237] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:23:55.650496 31594 raft_consensus.cc:1261] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:55.651889 32649 consensus_queue.cc:1035] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:55.652858 32594 raft_consensus.cc:1261] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:55.654487 32649 consensus_queue.cc:1035] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:55.656651 31502 raft_consensus.cc:1261] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:55.660264 32649 consensus_queue.cc:1035] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:55.663776 32658 raft_consensus.cc:2932] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.670447 31502 raft_consensus.cc:2932] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.670764 32594 raft_consensus.cc:2932] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.683768 31602 raft_consensus.cc:2932] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.686990 31357 catalog_manager.cc:5495] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:23:55.729166 31709 consensus_queue.cc:237] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:23:55.741811 31494 raft_consensus.cc:1261] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:23:55.743146 32594 raft_consensus.cc:1261] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:23:55.743642 32649 consensus_queue.cc:1035] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:55.746091 32708 consensus_queue.cc:1035] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:55.758944 32658 raft_consensus.cc:2932] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.762048 31502 raft_consensus.cc:2932] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.768165 32594 raft_consensus.cc:2932] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.771235 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 58a08ace2d684c14972d495f4a387a85 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:23:55.780357 31359 catalog_manager.cc:5495] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:23:55.785701 32513 raft_consensus.cc:1050] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:23:55.787802 32513 consensus_queue.cc:237] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:23:55.794124 31581 tablet_service.cc:1514] Processing DeleteTablet for tablet 58a08ace2d684c14972d495f4a387a85 with delete_type TABLET_DATA_TOMBSTONED (TS 516cb07ce3064ccbbca35e912d62816c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59024
I20260805 22:23:55.797901 32715 tablet_replica.cc:330] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c: stopping tablet replica
I20260805 22:23:55.798904 32715 raft_consensus.cc:2229] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:55.799654 32715 raft_consensus.cc:2258] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:55.803102 32715 ts_tablet_manager.cc:1905] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:23:55.805826 32594 raft_consensus.cc:1261] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:55.806422 31706 raft_consensus.cc:1261] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:55.807799 31594 raft_consensus.cc:1261] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:55.809830 32513 consensus_queue.cc:1035] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260805 22:23:55.810847 32674 consensus_queue.cc:1035] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:55.813158 32698 consensus_queue.cc:1035] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:55.848562 32715 ts_tablet_manager.cc:1918] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:23:55.849226 32715 log.cc:1196] T 58a08ace2d684c14972d495f4a387a85 P 516cb07ce3064ccbbca35e912d62816c: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-1-root/wals/58a08ace2d684c14972d495f4a387a85
I20260805 22:23:55.851081 31339 catalog_manager.cc:4841] TS 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): tablet 58a08ace2d684c14972d495f4a387a85 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:23:55.851233 32698 raft_consensus.cc:2932] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.853030 31706 raft_consensus.cc:2932] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.865713 32594 raft_consensus.cc:2932] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.872656 31594 raft_consensus.cc:2932] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.884582 31359 catalog_manager.cc:5495] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.907142 32575 raft_consensus.cc:1050] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:23:55.909821 32575 consensus_queue.cc:237] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:23:55.915007 31502 consensus_queue.cc:237] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:23:55.920518 32594 raft_consensus.cc:1261] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:55.922324 31502 raft_consensus.cc:1261] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:55.923197 32500 consensus_queue.cc:1035] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:55.922931 31706 raft_consensus.cc:1261] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:55.924789 32500 consensus_queue.cc:1035] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:55.926214 32594 raft_consensus.cc:1261] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:23:55.927924 31602 raft_consensus.cc:1261] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:23:55.928651 32575 consensus_queue.cc:1035] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:55.931913 32725 consensus_queue.cc:1035] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:55.934648 32722 consensus_queue.cc:1035] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:23:55.942296 32500 raft_consensus.cc:2932] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.943033 32725 raft_consensus.cc:2932] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.947281 32591 raft_consensus.cc:2932] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.950177 31602 raft_consensus.cc:2932] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.958508 32594 raft_consensus.cc:2932] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.959419 31709 raft_consensus.cc:2932] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.960770 31357 catalog_manager.cc:5495] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.964748 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet fe370453251f4300a8d90c397021595c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:23:55.967569 31358 catalog_manager.cc:5495] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:23:55.976864 31494 raft_consensus.cc:2932] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:55.981577 31698 tablet_service.cc:1514] Processing DeleteTablet for tablet fe370453251f4300a8d90c397021595c with delete_type TABLET_DATA_TOMBSTONED (TS feb5e7ab4b1541beaf2674c3c5500385 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56246
I20260805 22:23:55.988611 31604 consensus_queue.cc:237] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:23:55.999238 32739 tablet_replica.cc:330] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385: stopping tablet replica
I20260805 22:23:56.002290 31502 raft_consensus.cc:1261] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:23:56.000423 32739 raft_consensus.cc:2229] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:56.003213 32739 raft_consensus.cc:2258] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:56.003502 32639 consensus_queue.cc:1035] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:56.004880 32591 raft_consensus.cc:1261] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:23:56.007010 32739 ts_tablet_manager.cc:1905] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:23:56.018736 32735 consensus_queue.cc:1035] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:23:56.020215 32575 raft_consensus.cc:1050] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:23:56.029330 32739 ts_tablet_manager.cc:1918] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:23:56.030606 32739 log.cc:1196] T fe370453251f4300a8d90c397021595c P feb5e7ab4b1541beaf2674c3c5500385: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-2-root/wals/fe370453251f4300a8d90c397021595c
I20260805 22:23:56.033725 32575 consensus_queue.cc:237] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:23:56.035996 32500 raft_consensus.cc:2932] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:56.039893 31492 raft_consensus.cc:2932] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:56.033967 31339 catalog_manager.cc:4841] TS feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): tablet fe370453251f4300a8d90c397021595c (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:23:56.055291 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 78cb483d17014b3281316012dbb6f6e1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:23:56.054637 32594 raft_consensus.cc:2932] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:56.070108 31358 catalog_manager.cc:5495] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:56.073297 32591 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:56.080811 31706 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:56.081588 32639 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:23:56.085193 31698 tablet_service.cc:1514] Processing DeleteTablet for tablet 78cb483d17014b3281316012dbb6f6e1 with delete_type TABLET_DATA_TOMBSTONED (TS feb5e7ab4b1541beaf2674c3c5500385 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56246
I20260805 22:23:56.086282 32739 tablet_replica.cc:330] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385: stopping tablet replica
I20260805 22:23:56.086378 31502 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:56.088142 32739 raft_consensus.cc:2229] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:56.088925 32739 raft_consensus.cc:2258] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:56.088908 32741 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:23:56.091920 32741 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:56.094118 32739 ts_tablet_manager.cc:1905] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:23:56.106062 32575 raft_consensus.cc:2932] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:56.110060 31502 raft_consensus.cc:2932] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:56.121438 32594 raft_consensus.cc:2932] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:56.122638 32739 ts_tablet_manager.cc:1918] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:23:56.123142 32739 log.cc:1196] T 78cb483d17014b3281316012dbb6f6e1 P feb5e7ab4b1541beaf2674c3c5500385: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-2-root/wals/78cb483d17014b3281316012dbb6f6e1
I20260805 22:23:56.124818 31706 raft_consensus.cc:2932] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:56.163166 31339 catalog_manager.cc:4841] TS feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): tablet 78cb483d17014b3281316012dbb6f6e1 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:23:56.164956 31358 catalog_manager.cc:5495] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:56.338112 31594 consensus_queue.cc:237] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:23:56.341418 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 92975.2us
W20260805 22:23:56.363281 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 146896us
I20260805 22:23:56.364440 32594 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:23:56.384184 32741 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:23:56.403688 32594 raft_consensus.cc:2932] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
W20260805 22:23:56.406697 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 48509.7us
I20260805 22:23:56.407227 32741 raft_consensus.cc:2932] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:56.417858 31709 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:23:56.423314 32639 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:56.427731 31711 raft_consensus.cc:2932] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:56.434844 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 39e7b32cb2f646828a02f900ec2d47f1 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:23:56.445304 31359 catalog_manager.cc:5495] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:23:56.461676 31478 tablet_service.cc:1514] Processing DeleteTablet for tablet 39e7b32cb2f646828a02f900ec2d47f1 with delete_type TABLET_DATA_TOMBSTONED (TS f8e9f4c756da4203917c464e2b179994 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40714
I20260805 22:23:56.463816 32755 tablet_replica.cc:330] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994: stopping tablet replica
I20260805 22:23:56.466687 32755 raft_consensus.cc:2229] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:56.467725 32755 raft_consensus.cc:2258] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:56.472437 32755 ts_tablet_manager.cc:1905] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:23:56.486351 32755 ts_tablet_manager.cc:1918] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:23:56.486728 32755 log.cc:1196] T 39e7b32cb2f646828a02f900ec2d47f1 P f8e9f4c756da4203917c464e2b179994: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-0-root/wals/39e7b32cb2f646828a02f900ec2d47f1
I20260805 22:23:56.488312 31339 catalog_manager.cc:4841] TS f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): tablet 39e7b32cb2f646828a02f900ec2d47f1 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:23:57.522284 31596 consensus_queue.cc:237] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
W20260805 22:23:57.529219 31556 consensus_peers.cc:487] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: aee974c178e84c3d9a5dcfb31556a46f. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:57.530063 31502 raft_consensus.cc:1261] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:23:57.530174 31706 raft_consensus.cc:1261] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:23:57.533526 32735 consensus_queue.cc:1035] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:57.535145 32741 consensus_queue.cc:1035] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:57.543596 32735 raft_consensus.cc:2932] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:57.545611 31706 raft_consensus.cc:2932] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:57.550051 31502 raft_consensus.cc:2932] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:57.556464 31494 consensus_queue.cc:237] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:57.560798 31356 catalog_manager.cc:5495] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:57.563735 31596 raft_consensus.cc:1261] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:23:57.565212 32757 consensus_queue.cc:1035] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260805 22:23:57.566213 31447 consensus_peers.cc:487] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36ee93639c3f4084b94145c6ee60a2a8. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:57.565300 31706 raft_consensus.cc:1261] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:23:57.568214 32674 consensus_queue.cc:1035] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:23:57.586519 32753 raft_consensus.cc:2932] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:57.588312 31706 raft_consensus.cc:2932] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:57.598901 31602 raft_consensus.cc:2932] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:57.612509 31596 consensus_queue.cc:237] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:57.617717 31358 catalog_manager.cc:5495] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:57.630679 31706 raft_consensus.cc:1261] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260805 22:23:57.631168 31556 consensus_peers.cc:487] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dcea14dac2934d139e2c5ab6d0b8648d. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:57.630745 31502 raft_consensus.cc:1261] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:23:57.632270 32760 consensus_queue.cc:1035] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:23:57.636366 32760 consensus_queue.cc:1035] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:57.648016 32741 raft_consensus.cc:2932] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:57.649600 31502 raft_consensus.cc:2932] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:57.651978 31706 raft_consensus.cc:2932] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:57.663710 31502 consensus_queue.cc:237] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:57.668478 31359 catalog_manager.cc:5495] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:57.675137 31706 raft_consensus.cc:1261] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260805 22:23:57.675629 31447 consensus_peers.cc:487] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a1aa2dc5e5341f0a0450361d6c95814. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:23:57.676604 32757 consensus_queue.cc:1035] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:57.677747 31596 raft_consensus.cc:1261] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:23:57.679128 32757 consensus_queue.cc:1035] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:23:57.687419 32753 raft_consensus.cc:2932] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:57.691591 31596 raft_consensus.cc:2932] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:57.694931 31706 raft_consensus.cc:2932] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:23:57.709488 31358 catalog_manager.cc:5495] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260805 22:23:57.958663 316 ts_tablet_manager.cc:927] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:57.960825 316 tablet_copy_client.cc:316] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:41897
I20260805 22:23:57.962538 31623 tablet_copy_service.cc:140] P 516cb07ce3064ccbbca35e912d62816c: Received BeginTabletCopySession request for tablet aee974c178e84c3d9a5dcfb31556a46f from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:56384)
I20260805 22:23:57.963016 31623 tablet_copy_service.cc:161] P 516cb07ce3064ccbbca35e912d62816c: Beginning new tablet copy session on tablet aee974c178e84c3d9a5dcfb31556a46f from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:56384: session id = 6bf8c694a0ba40978195ae044a917caf-aee974c178e84c3d9a5dcfb31556a46f
I20260805 22:23:57.970626 31623 tablet_copy_source_session.cc:205] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:23:57.974195 316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aee974c178e84c3d9a5dcfb31556a46f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:57.989621 316 tablet_copy_client.cc:791] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:23:57.990242 316 tablet_copy_client.cc:655] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:23:57.994158 316 tablet_copy_client.cc:531] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:23:58.002559 316 tablet_bootstrap.cc:492] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:23:58.041834 316 tablet_bootstrap.cc:492] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:23:58.042949 316 tablet_bootstrap.cc:492] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:23:58.043762 316 ts_tablet_manager.cc:1397] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.041s user 0.016s sys 0.008s
I20260805 22:23:58.046506 319 ts_tablet_manager.cc:927] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:23:58.046902 316 raft_consensus.cc:348] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:58.047972 316 raft_consensus.cc:729] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:23:58.048379 319 tablet_copy_client.cc:316] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:45607
I20260805 22:23:58.048686 316 consensus_queue.cc:260] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:58.050433 31516 tablet_copy_service.cc:140] P f8e9f4c756da4203917c464e2b179994: Received BeginTabletCopySession request for tablet 36ee93639c3f4084b94145c6ee60a2a8 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:52098)
I20260805 22:23:58.051000 31516 tablet_copy_service.cc:161] P f8e9f4c756da4203917c464e2b179994: Beginning new tablet copy session on tablet 36ee93639c3f4084b94145c6ee60a2a8 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:52098: session id = 6bf8c694a0ba40978195ae044a917caf-36ee93639c3f4084b94145c6ee60a2a8
I20260805 22:23:58.057411 31516 tablet_copy_source_session.cc:205] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:23:58.058302 316 ts_tablet_manager.cc:1428] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260805 22:23:58.062130 31623 tablet_copy_service.cc:342] P 516cb07ce3064ccbbca35e912d62816c: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-aee974c178e84c3d9a5dcfb31556a46f received from {username='slave'} at 127.0.0.1:56384
I20260805 22:23:58.062454 319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36ee93639c3f4084b94145c6ee60a2a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:58.062541 31623 tablet_copy_service.cc:433] P 516cb07ce3064ccbbca35e912d62816c: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-aee974c178e84c3d9a5dcfb31556a46f on tablet aee974c178e84c3d9a5dcfb31556a46f with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:23:58.079913 319 tablet_copy_client.cc:791] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:23:58.080729 319 tablet_copy_client.cc:655] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:23:58.084228 319 tablet_copy_client.cc:531] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:23:58.092563 319 tablet_bootstrap.cc:492] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:23:58.109994 319 tablet_bootstrap.cc:492] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:23:58.110714 316 ts_tablet_manager.cc:927] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:23:58.110952 319 tablet_bootstrap.cc:492] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:23:58.111621 319 ts_tablet_manager.cc:1397] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.004s
I20260805 22:23:58.113777 316 tablet_copy_client.cc:316] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:41897
I20260805 22:23:58.113771 319 raft_consensus.cc:348] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:58.115271 31623 tablet_copy_service.cc:140] P 516cb07ce3064ccbbca35e912d62816c: Received BeginTabletCopySession request for tablet dcea14dac2934d139e2c5ab6d0b8648d from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:56384)
I20260805 22:23:58.115281 319 raft_consensus.cc:729] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:23:58.115724 31623 tablet_copy_service.cc:161] P 516cb07ce3064ccbbca35e912d62816c: Beginning new tablet copy session on tablet dcea14dac2934d139e2c5ab6d0b8648d from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:56384: session id = 6bf8c694a0ba40978195ae044a917caf-dcea14dac2934d139e2c5ab6d0b8648d
I20260805 22:23:58.116354 319 consensus_queue.cc:260] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:58.118643 319 ts_tablet_manager.cc:1428] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260805 22:23:58.121635 31516 tablet_copy_service.cc:342] P f8e9f4c756da4203917c464e2b179994: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-36ee93639c3f4084b94145c6ee60a2a8 received from {username='slave'} at 127.0.0.1:52098
I20260805 22:23:58.122027 31516 tablet_copy_service.cc:433] P f8e9f4c756da4203917c464e2b179994: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-36ee93639c3f4084b94145c6ee60a2a8 on tablet 36ee93639c3f4084b94145c6ee60a2a8 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:23:58.126679 319 ts_tablet_manager.cc:927] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:23:58.127039 31623 tablet_copy_source_session.cc:205] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:23:58.128235 319 tablet_copy_client.cc:316] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:45607
I20260805 22:23:58.131409 316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcea14dac2934d139e2c5ab6d0b8648d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:58.133653 31516 tablet_copy_service.cc:140] P f8e9f4c756da4203917c464e2b179994: Received BeginTabletCopySession request for tablet 6a1aa2dc5e5341f0a0450361d6c95814 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:52098)
I20260805 22:23:58.134099 31516 tablet_copy_service.cc:161] P f8e9f4c756da4203917c464e2b179994: Beginning new tablet copy session on tablet 6a1aa2dc5e5341f0a0450361d6c95814 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:52098: session id = 6bf8c694a0ba40978195ae044a917caf-6a1aa2dc5e5341f0a0450361d6c95814
I20260805 22:23:58.141651 31516 tablet_copy_source_session.cc:205] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:23:58.145730 319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a1aa2dc5e5341f0a0450361d6c95814. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:23:58.146926 316 tablet_copy_client.cc:791] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:23:58.147694 316 tablet_copy_client.cc:655] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:23:58.151394 316 tablet_copy_client.cc:531] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:23:58.157948 316 tablet_bootstrap.cc:492] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:23:58.161710 319 tablet_copy_client.cc:791] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:23:58.162346 319 tablet_copy_client.cc:655] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:23:58.165786 319 tablet_copy_client.cc:531] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:23:58.176232 319 tablet_bootstrap.cc:492] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:23:58.177836 316 tablet_bootstrap.cc:492] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:23:58.178689 316 tablet_bootstrap.cc:492] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:23:58.179291 316 ts_tablet_manager.cc:1397] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.008s
I20260805 22:23:58.182282 316 raft_consensus.cc:348] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:58.183053 316 raft_consensus.cc:729] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:23:58.183554 316 consensus_queue.cc:260] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:58.185585 316 ts_tablet_manager.cc:1428] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
I20260805 22:23:58.187354 31623 tablet_copy_service.cc:342] P 516cb07ce3064ccbbca35e912d62816c: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-dcea14dac2934d139e2c5ab6d0b8648d received from {username='slave'} at 127.0.0.1:56384
I20260805 22:23:58.187737 31623 tablet_copy_service.cc:433] P 516cb07ce3064ccbbca35e912d62816c: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-dcea14dac2934d139e2c5ab6d0b8648d on tablet dcea14dac2934d139e2c5ab6d0b8648d with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:23:58.197925 319 tablet_bootstrap.cc:492] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:23:58.198823 319 tablet_bootstrap.cc:492] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:23:58.199470 319 ts_tablet_manager.cc:1397] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.002s
I20260805 22:23:58.201959 319 raft_consensus.cc:348] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:58.202703 319 raft_consensus.cc:729] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:23:58.203212 319 consensus_queue.cc:260] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:23:58.205461 319 ts_tablet_manager.cc:1428] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260805 22:23:58.206949 31516 tablet_copy_service.cc:342] P f8e9f4c756da4203917c464e2b179994: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-6a1aa2dc5e5341f0a0450361d6c95814 received from {username='slave'} at 127.0.0.1:52098
I20260805 22:23:58.207290 31516 tablet_copy_service.cc:433] P f8e9f4c756da4203917c464e2b179994: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-6a1aa2dc5e5341f0a0450361d6c95814 on tablet 6a1aa2dc5e5341f0a0450361d6c95814 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:23:58.462595 32594 raft_consensus.cc:1203] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:23:58.543164 32594 raft_consensus.cc:1203] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:23:58.605074 32594 raft_consensus.cc:1203] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:23:58.690441 32594 raft_consensus.cc:1203] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:23:58.975853 32764 raft_consensus.cc:1050] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:23:58.977819 32764 consensus_queue.cc:237] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:23:58.987880 32594 raft_consensus.cc:1261] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:58.988346 31494 raft_consensus.cc:1261] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:58.989400 32764 consensus_queue.cc:1035] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:58.989665 31706 raft_consensus.cc:1261] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:58.990923 32741 consensus_queue.cc:1035] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:58.994028 32764 consensus_queue.cc:1035] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:59.015467 32735 raft_consensus.cc:2932] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.017417 31706 raft_consensus.cc:2932] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.020565 32594 raft_consensus.cc:2932] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.025590 32757 raft_consensus.cc:1050] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:23:59.027969 32757 consensus_queue.cc:237] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:23:59.027824 31495 raft_consensus.cc:2932] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.040336 31356 catalog_manager.cc:5495] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.056653 334 raft_consensus.cc:1050] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:23:59.059250 334 consensus_queue.cc:237] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:23:59.083887 32594 raft_consensus.cc:1261] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:59.086189 31604 raft_consensus.cc:1261] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:59.088644 32674 consensus_queue.cc:1035] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:59.091230 347 consensus_queue.cc:1035] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:59.092454 31706 raft_consensus.cc:1261] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:59.098912 32594 raft_consensus.cc:1261] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:59.100395 32674 consensus_queue.cc:1035] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:59.101781 31604 raft_consensus.cc:1261] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:59.102421 32674 consensus_queue.cc:1035] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:59.106448 32753 consensus_queue.cc:1035] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:59.106490 31706 raft_consensus.cc:1261] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:59.113637 347 consensus_queue.cc:1035] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:59.151036 32757 raft_consensus.cc:2932] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.153574 32594 raft_consensus.cc:2932] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.161345 31602 consensus_queue.cc:237] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:23:59.154129 32591 raft_consensus.cc:2932] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.155804 347 raft_consensus.cc:2932] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.162883 31604 raft_consensus.cc:2932] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.173907 32589 raft_consensus.cc:1261] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:23:59.176051 32735 consensus_queue.cc:1035] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:59.173839 31711 raft_consensus.cc:1261] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:23:59.192211 31351 catalog_manager.cc:5495] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:23:59.200111 31358 catalog_manager.cc:5495] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.201586 32741 consensus_queue.cc:1035] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:59.204357 31604 raft_consensus.cc:2932] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.210933 361 raft_consensus.cc:2932] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.215488 31706 raft_consensus.cc:2932] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.210773 31711 raft_consensus.cc:2932] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.224223 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet aee974c178e84c3d9a5dcfb31556a46f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:23:59.230718 32735 raft_consensus.cc:1050] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:23:59.233253 32735 consensus_queue.cc:237] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:23:59.252056 32591 raft_consensus.cc:2932] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.259578 32594 raft_consensus.cc:1261] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:59.266302 361 consensus_queue.cc:1035] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260805 22:23:59.269644 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 68440.6us
I20260805 22:23:59.273551 31711 raft_consensus.cc:2932] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.287549 31492 consensus_queue.cc:237] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:23:59.291770 31494 consensus_queue.cc:237] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:23:59.293946 31706 raft_consensus.cc:1261] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:23:59.300886 32757 consensus_queue.cc:1035] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:59.293299 31359 catalog_manager.cc:5495] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.297015 32594 raft_consensus.cc:1261] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:23:59.303131 32757 consensus_queue.cc:1035] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:59.307886 31709 raft_consensus.cc:1261] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:59.310720 32764 consensus_queue.cc:1035] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:23:59.314327 31709 raft_consensus.cc:1261] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:23:59.316418 32757 consensus_queue.cc:1035] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:59.316937 32594 raft_consensus.cc:1261] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:23:59.318707 31494 raft_consensus.cc:1261] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:23:59.325712 32674 consensus_queue.cc:1035] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:23:59.326015 32757 raft_consensus.cc:2932] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.330050 365 consensus_queue.cc:1035] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:23:59.333948 359 raft_consensus.cc:2932] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.340355 32594 raft_consensus.cc:2932] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.342284 365 raft_consensus.cc:2932] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.361351 32589 raft_consensus.cc:2932] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.371064 31709 raft_consensus.cc:2932] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.346331 32591 raft_consensus.cc:2932] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.363627 31478 tablet_service.cc:1514] Processing DeleteTablet for tablet aee974c178e84c3d9a5dcfb31556a46f with delete_type TABLET_DATA_TOMBSTONED (TS f8e9f4c756da4203917c464e2b179994 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40714
I20260805 22:23:59.362964 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 36ee93639c3f4084b94145c6ee60a2a8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:23:59.378072 31351 catalog_manager.cc:5495] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.379371 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 6a1aa2dc5e5341f0a0450361d6c95814 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:23:59.390978 31706 raft_consensus.cc:2932] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.395816 31359 catalog_manager.cc:5495] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:23:59.396116 31711 raft_consensus.cc:2932] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.397516 31581 tablet_service.cc:1514] Processing DeleteTablet for tablet 36ee93639c3f4084b94145c6ee60a2a8 with delete_type TABLET_DATA_TOMBSTONED (TS 516cb07ce3064ccbbca35e912d62816c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59024
I20260805 22:23:59.401769 374 tablet_replica.cc:330] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c: stopping tablet replica
I20260805 22:23:59.402973 374 raft_consensus.cc:2229] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:59.403658 374 raft_consensus.cc:2258] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:59.408226 374 ts_tablet_manager.cc:1905] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:23:59.414587 373 tablet_replica.cc:330] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994: stopping tablet replica
I20260805 22:23:59.414425 31351 catalog_manager.cc:5495] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.420843 31596 consensus_queue.cc:237] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:23:59.437148 374 ts_tablet_manager.cc:1918] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:23:59.437719 374 log.cc:1196] T 36ee93639c3f4084b94145c6ee60a2a8 P 516cb07ce3064ccbbca35e912d62816c: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-1-root/wals/36ee93639c3f4084b94145c6ee60a2a8
I20260805 22:23:59.440352 31339 catalog_manager.cc:4841] TS 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): tablet 36ee93639c3f4084b94145c6ee60a2a8 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:23:59.440786 31581 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a1aa2dc5e5341f0a0450361d6c95814 with delete_type TABLET_DATA_TOMBSTONED (TS 516cb07ce3064ccbbca35e912d62816c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59024
I20260805 22:23:59.442665 32589 raft_consensus.cc:1261] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:23:59.446275 374 tablet_replica.cc:330] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c: stopping tablet replica
I20260805 22:23:59.448318 374 raft_consensus.cc:2229] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:59.446828 363 consensus_queue.cc:1035] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:23:59.452104 31711 raft_consensus.cc:1261] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:23:59.455308 374 raft_consensus.cc:2258] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:59.455154 362 consensus_queue.cc:1035] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:23:59.464866 362 raft_consensus.cc:2932] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.472503 374 ts_tablet_manager.cc:1905] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:23:59.476495 31706 raft_consensus.cc:2932] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.479312 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet dcea14dac2934d139e2c5ab6d0b8648d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:23:59.477494 32591 raft_consensus.cc:2932] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:23:59.490249 374 ts_tablet_manager.cc:1918] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:23:59.490814 374 log.cc:1196] T 6a1aa2dc5e5341f0a0450361d6c95814 P 516cb07ce3064ccbbca35e912d62816c: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-1-root/wals/6a1aa2dc5e5341f0a0450361d6c95814
I20260805 22:23:59.491405 31351 catalog_manager.cc:5495] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:23:59.492339 31339 catalog_manager.cc:4841] TS 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): tablet 6a1aa2dc5e5341f0a0450361d6c95814 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:23:59.492295 373 raft_consensus.cc:2229] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:59.493590 373 raft_consensus.cc:2258] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:59.496677 373 ts_tablet_manager.cc:1905] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:23:59.516718 373 ts_tablet_manager.cc:1918] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:23:59.517932 373 log.cc:1196] T aee974c178e84c3d9a5dcfb31556a46f P f8e9f4c756da4203917c464e2b179994: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-0-root/wals/aee974c178e84c3d9a5dcfb31556a46f
I20260805 22:23:59.519994 31478 tablet_service.cc:1514] Processing DeleteTablet for tablet dcea14dac2934d139e2c5ab6d0b8648d with delete_type TABLET_DATA_TOMBSTONED (TS f8e9f4c756da4203917c464e2b179994 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40714
I20260805 22:23:59.520180 31339 catalog_manager.cc:4841] TS f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): tablet aee974c178e84c3d9a5dcfb31556a46f (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:23:59.521046 373 tablet_replica.cc:330] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994: stopping tablet replica
I20260805 22:23:59.522199 373 raft_consensus.cc:2229] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:23:59.522827 373 pending_rounds.cc:62] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994: Trying to abort 1 pending ops.
I20260805 22:23:59.523123 373 pending_rounds.cc:68] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7315327546293440512 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } } }
I20260805 22:23:59.524685 373 raft_consensus.cc:2894] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20260805 22:23:59.525133 373 raft_consensus.cc:2866] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260805 22:23:59.525578 373 raft_consensus.cc:2258] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:23:59.528539 373 ts_tablet_manager.cc:1905] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:23:59.540050 373 ts_tablet_manager.cc:1918] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:23:59.540403 373 log.cc:1196] T dcea14dac2934d139e2c5ab6d0b8648d P f8e9f4c756da4203917c464e2b179994: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-0-root/wals/dcea14dac2934d139e2c5ab6d0b8648d
I20260805 22:23:59.541953 31339 catalog_manager.cc:4841] TS f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): tablet dcea14dac2934d139e2c5ab6d0b8648d (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:00.548025 31494 consensus_queue.cc:237] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:00.554334 31709 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:00.554469 31604 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:00.555778 32757 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:00.556553 359 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260805 22:24:00.557569 31447 consensus_peers.cc:487] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8513af2d574d45b4a6d02428543b1402. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:00.564564 381 raft_consensus.cc:2932] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:00.566646 31709 raft_consensus.cc:2932] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:00.568034 31604 raft_consensus.cc:2932] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:00.576615 31596 consensus_queue.cc:237] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:00.583482 31351 catalog_manager.cc:5495] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:00.595870 31709 raft_consensus.cc:1261] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:00.596203 31494 raft_consensus.cc:1261] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:00.599627 363 consensus_queue.cc:1035] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260805 22:24:00.602473 31556 consensus_peers.cc:487] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bd1c7254b2b74c10a4fb473ea633544d. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:00.604262 32764 consensus_queue.cc:1035] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:00.636745 394 raft_consensus.cc:2932] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:00.638895 31494 raft_consensus.cc:2932] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:00.659008 31494 consensus_queue.cc:237] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:00.660599 31709 raft_consensus.cc:2932] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:00.667783 31359 catalog_manager.cc:5495] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:00.678125 31596 raft_consensus.cc:1261] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:00.678723 31711 raft_consensus.cc:1261] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260805 22:24:00.679132 31447 consensus_peers.cc:487] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1115c6a30f64413ab24c9d215d58a2c7. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:00.681537 32757 consensus_queue.cc:1035] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:00.682547 391 consensus_queue.cc:1035] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:00.734745 32757 raft_consensus.cc:2932] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:00.736632 31709 raft_consensus.cc:2932] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:00.754007 31358 catalog_manager.cc:5495] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:00.756693 31594 raft_consensus.cc:2932] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:00.758665 31596 consensus_queue.cc:237] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:00.775025 31706 raft_consensus.cc:1261] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260805 22:24:00.776412 31556 consensus_peers.cc:487] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0191f03fa8984a40afc7ec1e44970c8e. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:00.780349 363 consensus_queue.cc:1035] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:00.781211 31502 raft_consensus.cc:1261] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:00.782562 394 consensus_queue.cc:1035] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:00.789042 363 raft_consensus.cc:2932] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:00.790907 31706 raft_consensus.cc:2932] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:00.811728 31358 catalog_manager.cc:5495] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 2 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:00.814622 31494 raft_consensus.cc:2932] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:00.956949 410 ts_tablet_manager.cc:927] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:24:00.958828 410 tablet_copy_client.cc:316] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:45607
I20260805 22:24:00.960162 31516 tablet_copy_service.cc:140] P f8e9f4c756da4203917c464e2b179994: Received BeginTabletCopySession request for tablet 8513af2d574d45b4a6d02428543b1402 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:52098)
I20260805 22:24:00.960695 31516 tablet_copy_service.cc:161] P f8e9f4c756da4203917c464e2b179994: Beginning new tablet copy session on tablet 8513af2d574d45b4a6d02428543b1402 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:52098: session id = 6bf8c694a0ba40978195ae044a917caf-8513af2d574d45b4a6d02428543b1402
I20260805 22:24:00.967664 31516 tablet_copy_source_session.cc:205] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:00.972795 410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8513af2d574d45b4a6d02428543b1402. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:00.989040 410 tablet_copy_client.cc:791] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:00.989668 410 tablet_copy_client.cc:655] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:00.994426 410 tablet_copy_client.cc:531] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:01.003328 410 tablet_bootstrap.cc:492] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:01.023542 410 tablet_bootstrap.cc:492] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:01.024297 410 tablet_bootstrap.cc:492] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:01.024976 410 ts_tablet_manager.cc:1397] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.011s
I20260805 22:24:01.027559 410 raft_consensus.cc:348] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:01.028409 410 raft_consensus.cc:729] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:01.029006 410 consensus_queue.cc:260] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:01.032208 410 ts_tablet_manager.cc:1428] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260805 22:24:01.035305 31516 tablet_copy_service.cc:342] P f8e9f4c756da4203917c464e2b179994: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-8513af2d574d45b4a6d02428543b1402 received from {username='slave'} at 127.0.0.1:52098
I20260805 22:24:01.035735 31516 tablet_copy_service.cc:433] P f8e9f4c756da4203917c464e2b179994: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-8513af2d574d45b4a6d02428543b1402 on tablet 8513af2d574d45b4a6d02428543b1402 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:01.080816 410 ts_tablet_manager.cc:927] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:24:01.082369 410 tablet_copy_client.cc:316] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:45607
I20260805 22:24:01.083773 31516 tablet_copy_service.cc:140] P f8e9f4c756da4203917c464e2b179994: Received BeginTabletCopySession request for tablet 1115c6a30f64413ab24c9d215d58a2c7 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:52098)
I20260805 22:24:01.084194 31516 tablet_copy_service.cc:161] P f8e9f4c756da4203917c464e2b179994: Beginning new tablet copy session on tablet 1115c6a30f64413ab24c9d215d58a2c7 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:52098: session id = 6bf8c694a0ba40978195ae044a917caf-1115c6a30f64413ab24c9d215d58a2c7
I20260805 22:24:01.090354 31516 tablet_copy_source_session.cc:205] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:01.093271 410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1115c6a30f64413ab24c9d215d58a2c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:01.107518 410 tablet_copy_client.cc:791] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:01.108186 410 tablet_copy_client.cc:655] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:01.111987 410 tablet_copy_client.cc:531] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:01.121745 410 tablet_bootstrap.cc:492] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:01.145809 410 tablet_bootstrap.cc:492] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:01.146692 410 tablet_bootstrap.cc:492] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:01.147353 410 ts_tablet_manager.cc:1397] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.000s
I20260805 22:24:01.150787 410 raft_consensus.cc:348] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:01.151535 410 raft_consensus.cc:729] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:01.152215 410 consensus_queue.cc:260] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:01.154582 410 ts_tablet_manager.cc:1428] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.007s user 0.003s sys 0.004s
I20260805 22:24:01.159242 31516 tablet_copy_service.cc:342] P f8e9f4c756da4203917c464e2b179994: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-1115c6a30f64413ab24c9d215d58a2c7 received from {username='slave'} at 127.0.0.1:52098
I20260805 22:24:01.159588 31516 tablet_copy_service.cc:433] P f8e9f4c756da4203917c464e2b179994: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-1115c6a30f64413ab24c9d215d58a2c7 on tablet 1115c6a30f64413ab24c9d215d58a2c7 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:01.206295 410 ts_tablet_manager.cc:927] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:01.207995 410 tablet_copy_client.cc:316] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:41897
I20260805 22:24:01.209501 31623 tablet_copy_service.cc:140] P 516cb07ce3064ccbbca35e912d62816c: Received BeginTabletCopySession request for tablet 0191f03fa8984a40afc7ec1e44970c8e from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:56384)
I20260805 22:24:01.210026 31623 tablet_copy_service.cc:161] P 516cb07ce3064ccbbca35e912d62816c: Beginning new tablet copy session on tablet 0191f03fa8984a40afc7ec1e44970c8e from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:56384: session id = 6bf8c694a0ba40978195ae044a917caf-0191f03fa8984a40afc7ec1e44970c8e
I20260805 22:24:01.219772 422 ts_tablet_manager.cc:927] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:01.219856 31623 tablet_copy_source_session.cc:205] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:01.223434 422 tablet_copy_client.cc:316] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:41897
I20260805 22:24:01.223905 410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0191f03fa8984a40afc7ec1e44970c8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:01.226564 31623 tablet_copy_service.cc:140] P 516cb07ce3064ccbbca35e912d62816c: Received BeginTabletCopySession request for tablet bd1c7254b2b74c10a4fb473ea633544d from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:56384)
I20260805 22:24:01.227034 31623 tablet_copy_service.cc:161] P 516cb07ce3064ccbbca35e912d62816c: Beginning new tablet copy session on tablet bd1c7254b2b74c10a4fb473ea633544d from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:56384: session id = 6bf8c694a0ba40978195ae044a917caf-bd1c7254b2b74c10a4fb473ea633544d
I20260805 22:24:01.236739 31623 tablet_copy_source_session.cc:205] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:01.240877 422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1c7254b2b74c10a4fb473ea633544d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:01.247969 410 tablet_copy_client.cc:791] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:01.248613 410 tablet_copy_client.cc:655] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:01.258647 410 tablet_copy_client.cc:531] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:01.262400 422 tablet_copy_client.cc:791] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:01.263075 422 tablet_copy_client.cc:655] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:01.267306 422 tablet_copy_client.cc:531] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:01.270967 410 tablet_bootstrap.cc:492] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:01.286127 422 tablet_bootstrap.cc:492] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:01.310354 422 tablet_bootstrap.cc:492] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:01.311343 422 tablet_bootstrap.cc:492] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:01.312090 422 ts_tablet_manager.cc:1397] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.026s user 0.014s sys 0.009s
I20260805 22:24:01.312101 410 tablet_bootstrap.cc:492] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:01.315076 422 raft_consensus.cc:348] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:01.315970 422 raft_consensus.cc:729] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:01.316437 410 tablet_bootstrap.cc:492] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:01.316668 422 consensus_queue.cc:260] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:01.318913 410 ts_tablet_manager.cc:1397] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.048s user 0.019s sys 0.004s
I20260805 22:24:01.322099 410 raft_consensus.cc:348] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:01.323222 410 raft_consensus.cc:729] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:01.323829 410 consensus_queue.cc:260] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:01.326258 410 ts_tablet_manager.cc:1428] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20260805 22:24:01.327728 31623 tablet_copy_service.cc:342] P 516cb07ce3064ccbbca35e912d62816c: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-0191f03fa8984a40afc7ec1e44970c8e received from {username='slave'} at 127.0.0.1:56384
I20260805 22:24:01.328114 31623 tablet_copy_service.cc:433] P 516cb07ce3064ccbbca35e912d62816c: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-0191f03fa8984a40afc7ec1e44970c8e on tablet 0191f03fa8984a40afc7ec1e44970c8e with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:01.332505 422 ts_tablet_manager.cc:1428] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20260805 22:24:01.343497 31623 tablet_copy_service.cc:342] P 516cb07ce3064ccbbca35e912d62816c: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-bd1c7254b2b74c10a4fb473ea633544d received from {username='slave'} at 127.0.0.1:56384
I20260805 22:24:01.343964 31623 tablet_copy_service.cc:433] P 516cb07ce3064ccbbca35e912d62816c: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-bd1c7254b2b74c10a4fb473ea633544d on tablet bd1c7254b2b74c10a4fb473ea633544d with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:01.490834 32589 raft_consensus.cc:1203] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:01.622933 32589 raft_consensus.cc:1203] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:01.657015 32589 raft_consensus.cc:1203] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:01.720891 32589 raft_consensus.cc:1203] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20260805 22:24:02.053995 359 raft_consensus.cc:1050] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:02.056106 359 consensus_queue.cc:237] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:02.061632 32589 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:02.062186 31709 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:02.063704 31602 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:02.063532 381 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:02.065032 359 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:02.068607 32757 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:02.086802 381 raft_consensus.cc:2932] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.094117 31709 raft_consensus.cc:2932] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.099470 31602 raft_consensus.cc:2932] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.106290 32589 raft_consensus.cc:2932] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.112501 393 raft_consensus.cc:1050] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:02.118616 31359 catalog_manager.cc:5495] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.114881 393 consensus_queue.cc:237] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:02.142428 31704 raft_consensus.cc:1261] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:02.143987 428 consensus_queue.cc:1035] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:02.156819 32589 raft_consensus.cc:1261] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:02.156819 31494 raft_consensus.cc:1261] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:02.158275 393 consensus_queue.cc:1035] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:02.159966 428 consensus_queue.cc:1035] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:02.173470 428 raft_consensus.cc:2932] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.176612 32591 raft_consensus.cc:2932] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.182734 31492 consensus_queue.cc:237] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:02.186123 31494 raft_consensus.cc:2932] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.191599 32594 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:02.192816 31594 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:02.193203 381 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:02.196141 381 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:02.204582 31704 raft_consensus.cc:2932] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.205777 32757 raft_consensus.cc:2932] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.208940 32594 raft_consensus.cc:2932] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.204499 31351 catalog_manager.cc:5495] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.222554 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 8513af2d574d45b4a6d02428543b1402 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:02.222263 31596 raft_consensus.cc:2932] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.233196 31604 consensus_queue.cc:237] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:02.242726 32594 raft_consensus.cc:1261] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:02.244341 393 consensus_queue.cc:1035] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:02.250496 31704 raft_consensus.cc:1261] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:02.251793 393 consensus_queue.cc:1035] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:02.254534 439 raft_consensus.cc:1050] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:02.256810 439 consensus_queue.cc:237] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:02.256211 31358 catalog_manager.cc:5495] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.300763 32589 raft_consensus.cc:2932] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.300419 31706 raft_consensus.cc:1261] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:02.263342 363 raft_consensus.cc:2932] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.315649 381 consensus_queue.cc:1035] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:02.272020 31709 raft_consensus.cc:2932] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.324407 429 raft_consensus.cc:1050] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:02.325837 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet bd1c7254b2b74c10a4fb473ea633544d with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:02.326871 429 consensus_queue.cc:237] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:02.344584 31359 catalog_manager.cc:5495] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.355968 31698 tablet_service.cc:1514] Processing DeleteTablet for tablet 8513af2d574d45b4a6d02428543b1402 with delete_type TABLET_DATA_TOMBSTONED (TS feb5e7ab4b1541beaf2674c3c5500385 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56246
I20260805 22:24:02.356870 31478 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1c7254b2b74c10a4fb473ea633544d with delete_type TABLET_DATA_TOMBSTONED (TS f8e9f4c756da4203917c464e2b179994 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40714
I20260805 22:24:02.301223 31594 raft_consensus.cc:1261] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:02.299787 32594 raft_consensus.cc:1261] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:02.364701 381 consensus_queue.cc:1035] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:02.368780 32757 consensus_queue.cc:1035] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:02.374598 32589 raft_consensus.cc:1261] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 2 LEARNER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260805 22:24:02.375979 457 consensus_queue.cc:1035] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:02.388495 461 tablet_replica.cc:330] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994: stopping tablet replica
I20260805 22:24:02.388216 359 raft_consensus.cc:2932] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.389647 461 raft_consensus.cc:2229] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:02.394376 459 tablet_replica.cc:330] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385: stopping tablet replica
I20260805 22:24:02.396158 31604 raft_consensus.cc:2932] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.399489 461 raft_consensus.cc:2258] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:02.402508 32594 raft_consensus.cc:2932] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.404527 31704 raft_consensus.cc:2932] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.409842 459 raft_consensus.cc:2229] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:02.410503 459 raft_consensus.cc:2258] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:02.414054 459 ts_tablet_manager.cc:1905] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:02.418215 31358 catalog_manager.cc:5495] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:02.423317 461 ts_tablet_manager.cc:1905] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20260805 22:24:02.425451 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 57570.5us
I20260805 22:24:02.429077 31704 raft_consensus.cc:1261] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260805 22:24:02.429690 31494 raft_consensus.cc:1261] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260805 22:24:02.436066 31494 consensus_queue.cc:237] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:02.440853 32594 raft_consensus.cc:1261] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:02.449061 461 ts_tablet_manager.cc:1918] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:02.449652 461 log.cc:1196] T bd1c7254b2b74c10a4fb473ea633544d P f8e9f4c756da4203917c464e2b179994: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-0-root/wals/bd1c7254b2b74c10a4fb473ea633544d
I20260805 22:24:02.449543 381 consensus_queue.cc:1035] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:24:02.452911 31339 catalog_manager.cc:4841] TS f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): tablet bd1c7254b2b74c10a4fb473ea633544d (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:02.457759 438 raft_consensus.cc:2932] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.459147 31704 raft_consensus.cc:1261] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:02.459380 32594 raft_consensus.cc:2932] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.460465 381 consensus_queue.cc:1035] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:02.464300 31704 raft_consensus.cc:2932] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.467519 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 1115c6a30f64413ab24c9d215d58a2c7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:02.471097 428 consensus_queue.cc:1035] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:02.473852 459 ts_tablet_manager.cc:1918] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:02.475116 459 log.cc:1196] T 8513af2d574d45b4a6d02428543b1402 P feb5e7ab4b1541beaf2674c3c5500385: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-2-root/wals/8513af2d574d45b4a6d02428543b1402
I20260805 22:24:02.476337 428 consensus_queue.cc:1035] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:02.483978 31339 catalog_manager.cc:4841] TS feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): tablet 8513af2d574d45b4a6d02428543b1402 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:02.486773 31359 catalog_manager.cc:5495] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.498911 31581 tablet_service.cc:1514] Processing DeleteTablet for tablet 1115c6a30f64413ab24c9d215d58a2c7 with delete_type TABLET_DATA_TOMBSTONED (TS 516cb07ce3064ccbbca35e912d62816c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59024
I20260805 22:24:02.504529 470 tablet_replica.cc:330] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c: stopping tablet replica
I20260805 22:24:02.505702 470 raft_consensus.cc:2229] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:02.506363 470 raft_consensus.cc:2258] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:02.520766 457 raft_consensus.cc:2932] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.524303 31711 raft_consensus.cc:2932] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.526839 470 ts_tablet_manager.cc:1905] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:02.524904 31494 raft_consensus.cc:2932] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.533502 32589 raft_consensus.cc:2932] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.542371 31359 catalog_manager.cc:5495] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:02.545931 470 ts_tablet_manager.cc:1918] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:02.546420 470 log.cc:1196] T 1115c6a30f64413ab24c9d215d58a2c7 P 516cb07ce3064ccbbca35e912d62816c: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-1-root/wals/1115c6a30f64413ab24c9d215d58a2c7
I20260805 22:24:02.551054 31339 catalog_manager.cc:4841] TS 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): tablet 1115c6a30f64413ab24c9d215d58a2c7 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:02.572377 31594 consensus_queue.cc:237] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:02.578346 32589 raft_consensus.cc:1261] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260805 22:24:02.579871 31492 raft_consensus.cc:1261] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260805 22:24:02.580979 429 consensus_queue.cc:1035] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:02.582124 428 consensus_queue.cc:1035] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:02.589576 428 raft_consensus.cc:2932] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.599269 32594 raft_consensus.cc:2932] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.600566 31492 raft_consensus.cc:2932] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.606817 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 0191f03fa8984a40afc7ec1e44970c8e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:02.616171 31356 catalog_manager.cc:5495] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New cstate: current_term: 2 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:02.628470 31698 tablet_service.cc:1514] Processing DeleteTablet for tablet 0191f03fa8984a40afc7ec1e44970c8e with delete_type TABLET_DATA_TOMBSTONED (TS feb5e7ab4b1541beaf2674c3c5500385 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56246
I20260805 22:24:02.629462 459 tablet_replica.cc:330] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385: stopping tablet replica
I20260805 22:24:02.630349 459 raft_consensus.cc:2229] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:02.630860 459 raft_consensus.cc:2258] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:02.633252 459 ts_tablet_manager.cc:1905] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:02.645928 459 ts_tablet_manager.cc:1918] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20260805 22:24:02.646248 459 log.cc:1196] T 0191f03fa8984a40afc7ec1e44970c8e P feb5e7ab4b1541beaf2674c3c5500385: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-2-root/wals/0191f03fa8984a40afc7ec1e44970c8e
I20260805 22:24:02.647712 31339 catalog_manager.cc:4841] TS feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): tablet 0191f03fa8984a40afc7ec1e44970c8e (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:03.683398 31602 consensus_queue.cc:237] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:03.688274 31711 raft_consensus.cc:1261] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:03.689955 31492 raft_consensus.cc:1261] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:03.690264 428 consensus_queue.cc:1035] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:03.691291 429 consensus_queue.cc:1035] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260805 22:24:03.694988 31556 consensus_peers.cc:487] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2d7039038a134310977029d3b3fb9fc9. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:03.713400 429 raft_consensus.cc:2932] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:03.715242 31492 raft_consensus.cc:2932] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:03.715852 31706 raft_consensus.cc:2932] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:03.726557 31711 consensus_queue.cc:237] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:03.735793 31359 catalog_manager.cc:5495] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:03.743570 31602 raft_consensus.cc:1261] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260805 22:24:03.743713 31668 consensus_peers.cc:487] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d8fd526bfd0047e49803dbf11f8eafc6. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:03.750219 339 consensus_queue.cc:1035] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260805 22:24:03.760414 31492 raft_consensus.cc:1261] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:03.761965 478 consensus_queue.cc:1035] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:03.767127 339 raft_consensus.cc:2932] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:03.768802 31602 raft_consensus.cc:2932] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:03.782264 31704 consensus_queue.cc:237] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:03.792619 31492 raft_consensus.cc:2932] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:03.804654 31351 catalog_manager.cc:5495] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:03.812327 31594 raft_consensus.cc:1261] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:03.814839 339 consensus_queue.cc:1035] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260805 22:24:03.821221 31668 consensus_peers.cc:487] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42324773ce144eb784e14693eecbee72. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:03.828042 31492 raft_consensus.cc:1261] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:03.829578 453 consensus_queue.cc:1035] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:03.832773 453 raft_consensus.cc:2932] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:03.834352 31594 raft_consensus.cc:2932] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:03.839564 31494 raft_consensus.cc:2932] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:03.853084 31502 consensus_queue.cc:237] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:03.855705 31356 catalog_manager.cc:5495] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:03.862049 31704 raft_consensus.cc:1261] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:03.862344 31594 raft_consensus.cc:1261] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260805 22:24:03.862939 31447 consensus_peers.cc:487] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 94d0d26905694943b7135a9b805cf106. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:03.863672 32757 consensus_queue.cc:1035] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:03.864811 489 consensus_queue.cc:1035] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260805 22:24:03.879865 32757 raft_consensus.cc:2932] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:03.881850 31594 raft_consensus.cc:2932] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:03.881419 31704 raft_consensus.cc:2932] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:03.901046 31356 catalog_manager.cc:5495] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:04.088558 499 ts_tablet_manager.cc:927] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:04.090677 499 tablet_copy_client.cc:316] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:41897
I20260805 22:24:04.093075 31623 tablet_copy_service.cc:140] P 516cb07ce3064ccbbca35e912d62816c: Received BeginTabletCopySession request for tablet 2d7039038a134310977029d3b3fb9fc9 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:56384)
I20260805 22:24:04.093538 31623 tablet_copy_service.cc:161] P 516cb07ce3064ccbbca35e912d62816c: Beginning new tablet copy session on tablet 2d7039038a134310977029d3b3fb9fc9 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:56384: session id = 6bf8c694a0ba40978195ae044a917caf-2d7039038a134310977029d3b3fb9fc9
I20260805 22:24:04.099457 31623 tablet_copy_source_session.cc:205] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:04.101975 499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d7039038a134310977029d3b3fb9fc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:04.116940 499 tablet_copy_client.cc:791] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:04.117607 499 tablet_copy_client.cc:655] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:04.122043 499 tablet_copy_client.cc:531] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:04.131655 499 tablet_bootstrap.cc:492] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:04.157361 499 tablet_bootstrap.cc:492] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:04.158185 499 tablet_bootstrap.cc:492] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:04.158797 499 ts_tablet_manager.cc:1397] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.027s user 0.025s sys 0.000s
I20260805 22:24:04.160936 499 raft_consensus.cc:348] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:04.161609 499 raft_consensus.cc:729] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:04.162179 499 consensus_queue.cc:260] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:04.165028 499 ts_tablet_manager.cc:1428] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260805 22:24:04.166625 31623 tablet_copy_service.cc:342] P 516cb07ce3064ccbbca35e912d62816c: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-2d7039038a134310977029d3b3fb9fc9 received from {username='slave'} at 127.0.0.1:56384
I20260805 22:24:04.166997 31623 tablet_copy_service.cc:433] P 516cb07ce3064ccbbca35e912d62816c: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-2d7039038a134310977029d3b3fb9fc9 on tablet 2d7039038a134310977029d3b3fb9fc9 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:04.286329 499 ts_tablet_manager.cc:927] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:04.290517 499 tablet_copy_client.cc:316] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:44831
I20260805 22:24:04.292443 31729 tablet_copy_service.cc:140] P feb5e7ab4b1541beaf2674c3c5500385: Received BeginTabletCopySession request for tablet d8fd526bfd0047e49803dbf11f8eafc6 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:58348)
I20260805 22:24:04.292996 31729 tablet_copy_service.cc:161] P feb5e7ab4b1541beaf2674c3c5500385: Beginning new tablet copy session on tablet d8fd526bfd0047e49803dbf11f8eafc6 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:58348: session id = 6bf8c694a0ba40978195ae044a917caf-d8fd526bfd0047e49803dbf11f8eafc6
I20260805 22:24:04.300491 31729 tablet_copy_source_session.cc:205] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:04.303085 499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8fd526bfd0047e49803dbf11f8eafc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:04.317762 499 tablet_copy_client.cc:791] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:04.318348 499 tablet_copy_client.cc:655] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:04.324047 499 tablet_copy_client.cc:531] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:04.332891 499 tablet_bootstrap.cc:492] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:04.351374 499 tablet_bootstrap.cc:492] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:04.352260 499 tablet_bootstrap.cc:492] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:04.352787 499 ts_tablet_manager.cc:1397] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20260805 22:24:04.355152 499 raft_consensus.cc:348] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:04.355967 499 raft_consensus.cc:729] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:04.356594 499 consensus_queue.cc:260] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:04.360167 499 ts_tablet_manager.cc:1428] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.007s user 0.003s sys 0.006s
I20260805 22:24:04.361600 31729 tablet_copy_service.cc:342] P feb5e7ab4b1541beaf2674c3c5500385: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-d8fd526bfd0047e49803dbf11f8eafc6 received from {username='slave'} at 127.0.0.1:58348
I20260805 22:24:04.361949 31729 tablet_copy_service.cc:433] P feb5e7ab4b1541beaf2674c3c5500385: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-d8fd526bfd0047e49803dbf11f8eafc6 on tablet d8fd526bfd0047e49803dbf11f8eafc6 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:04.394168 499 ts_tablet_manager.cc:927] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:04.395788 499 tablet_copy_client.cc:316] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:44831
I20260805 22:24:04.397145 31729 tablet_copy_service.cc:140] P feb5e7ab4b1541beaf2674c3c5500385: Received BeginTabletCopySession request for tablet 42324773ce144eb784e14693eecbee72 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:58348)
I20260805 22:24:04.397660 31729 tablet_copy_service.cc:161] P feb5e7ab4b1541beaf2674c3c5500385: Beginning new tablet copy session on tablet 42324773ce144eb784e14693eecbee72 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:58348: session id = 6bf8c694a0ba40978195ae044a917caf-42324773ce144eb784e14693eecbee72
I20260805 22:24:04.403816 31729 tablet_copy_source_session.cc:205] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:04.406602 499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42324773ce144eb784e14693eecbee72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:04.415849 512 ts_tablet_manager.cc:927] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:24:04.418967 512 tablet_copy_client.cc:316] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:45607
I20260805 22:24:04.420269 31516 tablet_copy_service.cc:140] P f8e9f4c756da4203917c464e2b179994: Received BeginTabletCopySession request for tablet 94d0d26905694943b7135a9b805cf106 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:52098)
I20260805 22:24:04.420737 31516 tablet_copy_service.cc:161] P f8e9f4c756da4203917c464e2b179994: Beginning new tablet copy session on tablet 94d0d26905694943b7135a9b805cf106 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:52098: session id = 6bf8c694a0ba40978195ae044a917caf-94d0d26905694943b7135a9b805cf106
I20260805 22:24:04.422992 499 tablet_copy_client.cc:791] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:04.423683 499 tablet_copy_client.cc:655] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:04.426620 31516 tablet_copy_source_session.cc:205] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:04.426731 499 tablet_copy_client.cc:531] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:04.432044 512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d0d26905694943b7135a9b805cf106. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:04.435272 499 tablet_bootstrap.cc:492] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:04.455950 512 tablet_copy_client.cc:791] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:04.456699 512 tablet_copy_client.cc:655] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:04.460139 512 tablet_copy_client.cc:531] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:04.468366 512 tablet_bootstrap.cc:492] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:04.470197 499 tablet_bootstrap.cc:492] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:04.471047 499 tablet_bootstrap.cc:492] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:04.471696 499 ts_tablet_manager.cc:1397] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.037s user 0.019s sys 0.005s
I20260805 22:24:04.474071 499 raft_consensus.cc:348] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:04.474855 499 raft_consensus.cc:729] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:04.475307 499 consensus_queue.cc:260] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:04.480345 499 ts_tablet_manager.cc:1428] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20260805 22:24:04.482123 31729 tablet_copy_service.cc:342] P feb5e7ab4b1541beaf2674c3c5500385: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-42324773ce144eb784e14693eecbee72 received from {username='slave'} at 127.0.0.1:58348
I20260805 22:24:04.482465 31729 tablet_copy_service.cc:433] P feb5e7ab4b1541beaf2674c3c5500385: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-42324773ce144eb784e14693eecbee72 on tablet 42324773ce144eb784e14693eecbee72 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:04.489302 512 tablet_bootstrap.cc:492] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:04.490255 512 tablet_bootstrap.cc:492] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:04.490931 512 ts_tablet_manager.cc:1397] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.010s
I20260805 22:24:04.493558 512 raft_consensus.cc:348] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:04.494462 512 raft_consensus.cc:729] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:04.495144 512 consensus_queue.cc:260] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:04.497568 512 ts_tablet_manager.cc:1428] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.006s user 0.005s sys 0.003s
I20260805 22:24:04.500697 31516 tablet_copy_service.cc:342] P f8e9f4c756da4203917c464e2b179994: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-94d0d26905694943b7135a9b805cf106 received from {username='slave'} at 127.0.0.1:52098
I20260805 22:24:04.501082 31516 tablet_copy_service.cc:433] P f8e9f4c756da4203917c464e2b179994: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-94d0d26905694943b7135a9b805cf106 on tablet 94d0d26905694943b7135a9b805cf106 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:04.514729 32594 raft_consensus.cc:1203] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:04.786283 32594 raft_consensus.cc:1203] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:04.810894 32594 raft_consensus.cc:1203] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:04.879359 472 raft_consensus.cc:1050] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:04.881757 472 consensus_queue.cc:237] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:04.888957 31706 raft_consensus.cc:1261] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:04.888993 32594 raft_consensus.cc:1261] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:04.890554 472 consensus_queue.cc:1035] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:04.891754 31492 raft_consensus.cc:1261] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:04.892836 428 consensus_queue.cc:1035] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:04.896273 428 consensus_queue.cc:1035] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:04.909941 429 raft_consensus.cc:2932] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:04.911808 32594 raft_consensus.cc:2932] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:04.923683 31359 catalog_manager.cc:5495] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:04.927698 31711 raft_consensus.cc:2932] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:04.927815 31492 raft_consensus.cc:2932] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:04.959661 32594 raft_consensus.cc:1203] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:04.988188 31596 consensus_queue.cc:237] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:04.993888 32594 raft_consensus.cc:1261] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:04.995697 429 consensus_queue.cc:1035] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:04.999989 31499 raft_consensus.cc:1261] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:05.002442 428 consensus_queue.cc:1035] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:05.004812 428 raft_consensus.cc:2932] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.014771 32594 raft_consensus.cc:2932] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.017288 31502 raft_consensus.cc:2932] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.027118 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 2d7039038a134310977029d3b3fb9fc9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:05.034437 31358 catalog_manager.cc:5495] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.045020 31698 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d7039038a134310977029d3b3fb9fc9 with delete_type TABLET_DATA_TOMBSTONED (TS feb5e7ab4b1541beaf2674c3c5500385 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56246
I20260805 22:24:05.056656 532 tablet_replica.cc:330] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385: stopping tablet replica
I20260805 22:24:05.057986 532 raft_consensus.cc:2229] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:05.058714 532 raft_consensus.cc:2258] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:05.062336 532 ts_tablet_manager.cc:1905] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:05.076779 532 ts_tablet_manager.cc:1918] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:05.077180 532 log.cc:1196] T 2d7039038a134310977029d3b3fb9fc9 P feb5e7ab4b1541beaf2674c3c5500385: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-2-root/wals/2d7039038a134310977029d3b3fb9fc9
I20260805 22:24:05.078665 31339 catalog_manager.cc:4841] TS feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): tablet 2d7039038a134310977029d3b3fb9fc9 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:05.214159 453 raft_consensus.cc:1050] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:05.216985 453 consensus_queue.cc:237] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:05.226598 516 raft_consensus.cc:1050] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:05.228881 516 consensus_queue.cc:237] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:05.232822 31596 raft_consensus.cc:1261] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:05.235563 32594 raft_consensus.cc:1261] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:05.237241 478 consensus_queue.cc:1035] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:05.240270 478 consensus_queue.cc:1035] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:05.247682 31492 raft_consensus.cc:1261] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:05.248163 453 raft_consensus.cc:2932] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.250975 537 consensus_queue.cc:1035] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:05.255124 32594 raft_consensus.cc:1261] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:05.257531 31602 raft_consensus.cc:2932] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.258962 543 consensus_queue.cc:1035] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:05.262241 31502 raft_consensus.cc:1261] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:05.268788 31596 raft_consensus.cc:1261] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:05.275754 537 consensus_queue.cc:1035] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:05.278072 537 consensus_queue.cc:1035] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:05.278724 32589 raft_consensus.cc:2932] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.288551 31492 raft_consensus.cc:2932] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.300909 31356 catalog_manager.cc:5495] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.307230 453 raft_consensus.cc:2932] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.313736 31596 raft_consensus.cc:2932] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.322343 32594 raft_consensus.cc:2932] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.328327 31711 consensus_queue.cc:237] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:05.331584 31502 raft_consensus.cc:2932] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.339391 31602 raft_consensus.cc:1261] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:05.340901 543 consensus_queue.cc:1035] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:05.350286 31351 catalog_manager.cc:5495] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.351534 545 raft_consensus.cc:2932] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.361001 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 42324773ce144eb784e14693eecbee72 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:05.367548 32591 raft_consensus.cc:1261] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:05.366941 31596 raft_consensus.cc:2932] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.384543 543 consensus_queue.cc:1035] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:05.389180 32591 raft_consensus.cc:2932] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.397286 31358 catalog_manager.cc:5495] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.411754 31478 tablet_service.cc:1514] Processing DeleteTablet for tablet 42324773ce144eb784e14693eecbee72 with delete_type TABLET_DATA_TOMBSTONED (TS f8e9f4c756da4203917c464e2b179994 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40714
I20260805 22:24:05.417819 555 tablet_replica.cc:330] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994: stopping tablet replica
I20260805 22:24:05.418926 555 raft_consensus.cc:2229] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:05.419602 555 raft_consensus.cc:2258] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:05.422899 555 ts_tablet_manager.cc:1905] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:05.426560 31711 consensus_queue.cc:237] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:05.430573 381 raft_consensus.cc:1050] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:05.432812 381 consensus_queue.cc:237] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:05.440950 32591 raft_consensus.cc:1261] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:05.442559 545 consensus_queue.cc:1035] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:05.442751 555 ts_tablet_manager.cc:1918] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:05.443416 555 log.cc:1196] T 42324773ce144eb784e14693eecbee72 P f8e9f4c756da4203917c464e2b179994: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-0-root/wals/42324773ce144eb784e14693eecbee72
I20260805 22:24:05.447605 32594 raft_consensus.cc:1261] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:05.448148 31596 raft_consensus.cc:1261] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:05.449903 32757 consensus_queue.cc:1035] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:05.450874 557 consensus_queue.cc:1035] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:05.450466 545 raft_consensus.cc:2932] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.452917 31711 raft_consensus.cc:1261] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:05.453612 31339 catalog_manager.cc:4841] TS f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): tablet 42324773ce144eb784e14693eecbee72 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:05.455371 32757 consensus_queue.cc:1035] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:05.454948 31502 raft_consensus.cc:1261] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:05.456058 32589 raft_consensus.cc:2932] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.462751 453 consensus_queue.cc:1035] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:24:05.468016 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d8fd526bfd0047e49803dbf11f8eafc6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:05.469259 31492 raft_consensus.cc:2932] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.476970 31356 catalog_manager.cc:5495] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.482056 381 raft_consensus.cc:2932] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.485771 31711 raft_consensus.cc:2932] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.489750 32591 raft_consensus.cc:2932] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.495311 31596 raft_consensus.cc:2932] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.505781 31351 catalog_manager.cc:5495] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.509614 31581 tablet_service.cc:1514] Processing DeleteTablet for tablet d8fd526bfd0047e49803dbf11f8eafc6 with delete_type TABLET_DATA_TOMBSTONED (TS 516cb07ce3064ccbbca35e912d62816c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59024
I20260805 22:24:05.520973 565 tablet_replica.cc:330] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c: stopping tablet replica
I20260805 22:24:05.522089 565 raft_consensus.cc:2229] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:05.522712 565 raft_consensus.cc:2258] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:05.526767 565 ts_tablet_manager.cc:1905] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:05.536885 31492 consensus_queue.cc:237] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:05.540766 31711 raft_consensus.cc:1261] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:05.541128 32591 raft_consensus.cc:1261] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:05.542222 557 consensus_queue.cc:1035] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:05.543085 381 consensus_queue.cc:1035] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:05.547639 565 ts_tablet_manager.cc:1918] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:05.549505 565 log.cc:1196] T d8fd526bfd0047e49803dbf11f8eafc6 P 516cb07ce3064ccbbca35e912d62816c: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-1-root/wals/d8fd526bfd0047e49803dbf11f8eafc6
I20260805 22:24:05.555220 31339 catalog_manager.cc:4841] TS 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): tablet d8fd526bfd0047e49803dbf11f8eafc6 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:05.554992 557 raft_consensus.cc:2932] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.558467 32591 raft_consensus.cc:2932] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.556990 31711 raft_consensus.cc:2932] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:05.565500 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 94d0d26905694943b7135a9b805cf106 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:05.581820 31351 catalog_manager.cc:5495] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:05.594126 31581 tablet_service.cc:1514] Processing DeleteTablet for tablet 94d0d26905694943b7135a9b805cf106 with delete_type TABLET_DATA_TOMBSTONED (TS 516cb07ce3064ccbbca35e912d62816c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59024
I20260805 22:24:05.595027 565 tablet_replica.cc:330] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c: stopping tablet replica
I20260805 22:24:05.602559 565 raft_consensus.cc:2229] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:05.603283 565 raft_consensus.cc:2258] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:05.606405 565 ts_tablet_manager.cc:1905] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:05.619980 565 ts_tablet_manager.cc:1918] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:05.620291 565 log.cc:1196] T 94d0d26905694943b7135a9b805cf106 P 516cb07ce3064ccbbca35e912d62816c: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-1-root/wals/94d0d26905694943b7135a9b805cf106
I20260805 22:24:05.621582 31339 catalog_manager.cc:4841] TS 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): tablet 94d0d26905694943b7135a9b805cf106 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:06.640262 31596 consensus_queue.cc:237] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:06.647419 31492 raft_consensus.cc:1261] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260805 22:24:06.647850 31556 consensus_peers.cc:487] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3f76fdd524184682ae27abc293a7fff6. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:06.648900 31706 raft_consensus.cc:1261] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:06.650197 429 consensus_queue.cc:1035] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:06.651423 531 consensus_queue.cc:1035] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:06.672387 566 raft_consensus.cc:2932] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:06.678570 31711 raft_consensus.cc:2932] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:06.689845 31492 consensus_queue.cc:237] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
W20260805 22:24:06.699488 31447 consensus_peers.cc:487] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3095cdb25afc4c26bd9d1e03a919b474. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:06.701514 31706 raft_consensus.cc:1261] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:06.701613 31594 raft_consensus.cc:1261] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:06.704739 31502 raft_consensus.cc:2932] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:06.697031 31358 catalog_manager.cc:5495] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:06.708753 557 consensus_queue.cc:1035] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:06.715871 381 consensus_queue.cc:1035] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:06.740082 489 raft_consensus.cc:2932] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:06.751412 31594 raft_consensus.cc:2932] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:06.757292 31706 raft_consensus.cc:2932] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:06.762095 31596 consensus_queue.cc:237] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:06.776597 31356 catalog_manager.cc:5495] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:06.796638 31502 raft_consensus.cc:1261] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260805 22:24:06.797250 31556 consensus_peers.cc:487] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f0eb14e6bb5c4855859507799f2b9e16. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:06.797521 31706 raft_consensus.cc:1261] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:06.797998 531 consensus_queue.cc:1035] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:06.799010 572 consensus_queue.cc:1035] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:06.806522 531 raft_consensus.cc:2932] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:06.808045 31711 raft_consensus.cc:2932] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:06.809053 31502 raft_consensus.cc:2932] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:06.824707 31358 catalog_manager.cc:5495] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:06.835217 31502 consensus_queue.cc:237] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:06.841118 31706 raft_consensus.cc:1261] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260805 22:24:06.841605 31447 consensus_peers.cc:487] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d68a856424b4ceb8f4f54b4050e8dc4. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:06.842407 381 consensus_queue.cc:1035] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:06.843699 31596 raft_consensus.cc:1261] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:06.845809 381 consensus_queue.cc:1035] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:06.850826 489 raft_consensus.cc:2932] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:06.852612 31706 raft_consensus.cc:2932] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:06.863507 31594 raft_consensus.cc:2932] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:06.874691 31356 catalog_manager.cc:5495] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:07.027846 592 ts_tablet_manager.cc:927] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:07.029762 592 tablet_copy_client.cc:316] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:41897
I20260805 22:24:07.031510 31623 tablet_copy_service.cc:140] P 516cb07ce3064ccbbca35e912d62816c: Received BeginTabletCopySession request for tablet 3f76fdd524184682ae27abc293a7fff6 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:56384)
I20260805 22:24:07.031919 31623 tablet_copy_service.cc:161] P 516cb07ce3064ccbbca35e912d62816c: Beginning new tablet copy session on tablet 3f76fdd524184682ae27abc293a7fff6 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:56384: session id = 6bf8c694a0ba40978195ae044a917caf-3f76fdd524184682ae27abc293a7fff6
I20260805 22:24:07.039736 31623 tablet_copy_source_session.cc:205] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:07.044602 592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3f76fdd524184682ae27abc293a7fff6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:07.059818 592 tablet_copy_client.cc:791] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:07.060458 592 tablet_copy_client.cc:655] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:07.064502 592 tablet_copy_client.cc:531] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:07.071731 592 tablet_bootstrap.cc:492] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:07.090109 592 tablet_bootstrap.cc:492] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:07.091080 592 tablet_bootstrap.cc:492] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:07.091799 592 ts_tablet_manager.cc:1397] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.009s
I20260805 22:24:07.094944 592 raft_consensus.cc:348] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:07.095793 592 raft_consensus.cc:729] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:07.096459 592 consensus_queue.cc:260] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:07.099695 592 ts_tablet_manager.cc:1428] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260805 22:24:07.101470 31623 tablet_copy_service.cc:342] P 516cb07ce3064ccbbca35e912d62816c: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-3f76fdd524184682ae27abc293a7fff6 received from {username='slave'} at 127.0.0.1:56384
I20260805 22:24:07.101899 31623 tablet_copy_service.cc:433] P 516cb07ce3064ccbbca35e912d62816c: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-3f76fdd524184682ae27abc293a7fff6 on tablet 3f76fdd524184682ae27abc293a7fff6 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:07.225466 592 ts_tablet_manager.cc:927] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:24:07.227257 592 tablet_copy_client.cc:316] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:45607
I20260805 22:24:07.228889 31516 tablet_copy_service.cc:140] P f8e9f4c756da4203917c464e2b179994: Received BeginTabletCopySession request for tablet 3095cdb25afc4c26bd9d1e03a919b474 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:52098)
I20260805 22:24:07.229497 31516 tablet_copy_service.cc:161] P f8e9f4c756da4203917c464e2b179994: Beginning new tablet copy session on tablet 3095cdb25afc4c26bd9d1e03a919b474 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:52098: session id = 6bf8c694a0ba40978195ae044a917caf-3095cdb25afc4c26bd9d1e03a919b474
I20260805 22:24:07.239651 31516 tablet_copy_source_session.cc:205] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:07.242497 592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3095cdb25afc4c26bd9d1e03a919b474. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:07.257771 592 tablet_copy_client.cc:791] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:07.258399 592 tablet_copy_client.cc:655] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:07.270937 592 tablet_copy_client.cc:531] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:07.286829 592 tablet_bootstrap.cc:492] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:07.295251 599 ts_tablet_manager.cc:927] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:24:07.308265 599 tablet_copy_client.cc:316] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:45607
I20260805 22:24:07.311270 31516 tablet_copy_service.cc:140] P f8e9f4c756da4203917c464e2b179994: Received BeginTabletCopySession request for tablet 9d68a856424b4ceb8f4f54b4050e8dc4 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:52098)
I20260805 22:24:07.311702 31516 tablet_copy_service.cc:161] P f8e9f4c756da4203917c464e2b179994: Beginning new tablet copy session on tablet 9d68a856424b4ceb8f4f54b4050e8dc4 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:52098: session id = 6bf8c694a0ba40978195ae044a917caf-9d68a856424b4ceb8f4f54b4050e8dc4
I20260805 22:24:07.314572 592 tablet_bootstrap.cc:492] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:07.315299 592 tablet_bootstrap.cc:492] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:07.315886 592 ts_tablet_manager.cc:1397] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.029s user 0.009s sys 0.010s
I20260805 22:24:07.319286 592 raft_consensus.cc:348] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:07.319973 592 raft_consensus.cc:729] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:07.319900 31516 tablet_copy_source_session.cc:205] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:07.320577 592 consensus_queue.cc:260] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:07.324404 592 ts_tablet_manager.cc:1428] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20260805 22:24:07.324729 599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d68a856424b4ceb8f4f54b4050e8dc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:07.326162 31516 tablet_copy_service.cc:342] P f8e9f4c756da4203917c464e2b179994: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-3095cdb25afc4c26bd9d1e03a919b474 received from {username='slave'} at 127.0.0.1:52098
I20260805 22:24:07.326655 31516 tablet_copy_service.cc:433] P f8e9f4c756da4203917c464e2b179994: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-3095cdb25afc4c26bd9d1e03a919b474 on tablet 3095cdb25afc4c26bd9d1e03a919b474 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:07.346941 592 ts_tablet_manager.cc:927] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:07.347975 599 tablet_copy_client.cc:791] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:07.348824 599 tablet_copy_client.cc:655] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:07.349967 592 tablet_copy_client.cc:316] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:41897
I20260805 22:24:07.353955 31623 tablet_copy_service.cc:140] P 516cb07ce3064ccbbca35e912d62816c: Received BeginTabletCopySession request for tablet f0eb14e6bb5c4855859507799f2b9e16 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:56384)
I20260805 22:24:07.354517 31623 tablet_copy_service.cc:161] P 516cb07ce3064ccbbca35e912d62816c: Beginning new tablet copy session on tablet f0eb14e6bb5c4855859507799f2b9e16 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:56384: session id = 6bf8c694a0ba40978195ae044a917caf-f0eb14e6bb5c4855859507799f2b9e16
I20260805 22:24:07.355050 599 tablet_copy_client.cc:531] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:07.364682 599 tablet_bootstrap.cc:492] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:07.365495 31623 tablet_copy_source_session.cc:205] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:07.370642 592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f0eb14e6bb5c4855859507799f2b9e16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:07.387188 592 tablet_copy_client.cc:791] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:07.387840 592 tablet_copy_client.cc:655] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:07.391618 592 tablet_copy_client.cc:531] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:07.391896 599 tablet_bootstrap.cc:492] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:07.392776 599 tablet_bootstrap.cc:492] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:07.393419 599 ts_tablet_manager.cc:1397] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.029s user 0.018s sys 0.007s
I20260805 22:24:07.395855 599 raft_consensus.cc:348] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:07.396464 599 raft_consensus.cc:729] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:07.397101 599 consensus_queue.cc:260] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:07.400079 599 ts_tablet_manager.cc:1428] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.006s user 0.001s sys 0.004s
I20260805 22:24:07.401518 31516 tablet_copy_service.cc:342] P f8e9f4c756da4203917c464e2b179994: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-9d68a856424b4ceb8f4f54b4050e8dc4 received from {username='slave'} at 127.0.0.1:52098
I20260805 22:24:07.401942 31516 tablet_copy_service.cc:433] P f8e9f4c756da4203917c464e2b179994: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-9d68a856424b4ceb8f4f54b4050e8dc4 on tablet 9d68a856424b4ceb8f4f54b4050e8dc4 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:07.404596 592 tablet_bootstrap.cc:492] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:07.426544 592 tablet_bootstrap.cc:492] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:07.427433 592 tablet_bootstrap.cc:492] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:07.428112 592 ts_tablet_manager.cc:1397] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.024s user 0.018s sys 0.000s
I20260805 22:24:07.429937 592 raft_consensus.cc:348] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:07.430529 592 raft_consensus.cc:729] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:07.430968 592 consensus_queue.cc:260] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:07.432713 592 ts_tablet_manager.cc:1428] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260805 22:24:07.434093 31623 tablet_copy_service.cc:342] P 516cb07ce3064ccbbca35e912d62816c: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-f0eb14e6bb5c4855859507799f2b9e16 received from {username='slave'} at 127.0.0.1:56384
I20260805 22:24:07.434398 31623 tablet_copy_service.cc:433] P 516cb07ce3064ccbbca35e912d62816c: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-f0eb14e6bb5c4855859507799f2b9e16 on tablet f0eb14e6bb5c4855859507799f2b9e16 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:07.635092 32589 raft_consensus.cc:1203] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:07.729532 32589 raft_consensus.cc:1203] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:07.749802 32589 raft_consensus.cc:1203] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:07.845125 32589 raft_consensus.cc:1203] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:08.109978 489 raft_consensus.cc:1050] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:08.112124 489 consensus_queue.cc:237] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:08.117654 31711 raft_consensus.cc:1261] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:08.117929 31596 raft_consensus.cc:1261] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:08.117942 32589 raft_consensus.cc:1261] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:08.119022 607 consensus_queue.cc:1035] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:08.119943 611 consensus_queue.cc:1035] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260805 22:24:08.127907 381 consensus_queue.cc:1035] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:08.143877 611 raft_consensus.cc:2932] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.147117 31711 raft_consensus.cc:2932] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.151232 31596 raft_consensus.cc:2932] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.159955 32589 raft_consensus.cc:2932] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.171191 429 raft_consensus.cc:1050] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:08.173488 429 consensus_queue.cc:237] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:08.164942 31358 catalog_manager.cc:5495] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:08.181687 31706 raft_consensus.cc:1261] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:08.182377 32591 raft_consensus.cc:1261] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:08.183485 31502 raft_consensus.cc:1261] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:08.185228 569 consensus_queue.cc:1035] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:08.197573 429 consensus_queue.cc:1035] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:08.198658 608 consensus_queue.cc:1035] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:08.213905 608 raft_consensus.cc:1050] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:08.225107 531 raft_consensus.cc:2932] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.226473 608 consensus_queue.cc:237] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:08.234592 31502 raft_consensus.cc:2932] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.239277 32589 raft_consensus.cc:2932] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.249197 31711 raft_consensus.cc:1261] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:08.251370 569 consensus_queue.cc:1035] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:08.255106 381 raft_consensus.cc:1050] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:08.256369 31711 raft_consensus.cc:2932] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.260275 32589 raft_consensus.cc:1261] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:08.264539 608 consensus_queue.cc:1035] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:08.267109 31492 raft_consensus.cc:1261] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:08.268630 381 consensus_queue.cc:237] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:08.273584 608 consensus_queue.cc:1035] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20260805 22:24:08.278993 31596 raft_consensus.cc:1261] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:08.280331 32591 raft_consensus.cc:1261] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:08.281669 381 consensus_queue.cc:1035] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:08.287210 611 consensus_queue.cc:1035] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:08.298483 31706 raft_consensus.cc:1261] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:08.306471 611 consensus_queue.cc:1035] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.005s
I20260805 22:24:08.335171 381 raft_consensus.cc:2932] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.342447 31709 raft_consensus.cc:2932] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.351382 632 raft_consensus.cc:2932] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.354426 31494 raft_consensus.cc:2932] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.360417 32591 raft_consensus.cc:2932] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.362660 31711 raft_consensus.cc:2932] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.385180 31594 raft_consensus.cc:2932] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.391813 31351 catalog_manager.cc:5495] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.393536 31351 catalog_manager.cc:5495] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.404568 31492 consensus_queue.cc:237] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:08.401367 32591 raft_consensus.cc:2932] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.418275 31709 raft_consensus.cc:1261] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:08.419809 611 consensus_queue.cc:1035] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:08.421583 32591 raft_consensus.cc:1261] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:08.425523 611 consensus_queue.cc:1035] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:08.426064 31358 catalog_manager.cc:5495] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.431377 614 raft_consensus.cc:2932] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.433117 31711 raft_consensus.cc:2932] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.439445 31602 consensus_queue.cc:237] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:08.446322 32591 raft_consensus.cc:1261] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:08.447786 31596 consensus_queue.cc:237] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:08.449545 531 consensus_queue.cc:1035] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:08.452159 31709 raft_consensus.cc:1261] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:08.453418 32589 raft_consensus.cc:2932] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.461656 31711 raft_consensus.cc:1261] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:08.464609 569 consensus_queue.cc:1035] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:08.466372 569 consensus_queue.cc:1035] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:08.473660 32594 raft_consensus.cc:1261] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:08.477275 531 raft_consensus.cc:2932] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.485181 632 raft_consensus.cc:2932] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.487202 429 consensus_queue.cc:1035] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:08.478678 31709 raft_consensus.cc:2932] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.492801 32589 raft_consensus.cc:2932] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.494829 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f0eb14e6bb5c4855859507799f2b9e16 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:08.495738 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 9d68a856424b4ceb8f4f54b4050e8dc4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:08.504721 31494 consensus_queue.cc:237] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:08.508939 31706 raft_consensus.cc:2932] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.514129 32591 raft_consensus.cc:2932] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.517273 31358 catalog_manager.cc:5495] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.536386 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3f76fdd524184682ae27abc293a7fff6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:08.536834 32589 raft_consensus.cc:1261] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:08.538421 614 consensus_queue.cc:1035] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:08.541747 31357 catalog_manager.cc:5495] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:08.544303 31596 raft_consensus.cc:1261] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:08.546101 31581 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d68a856424b4ceb8f4f54b4050e8dc4 with delete_type TABLET_DATA_TOMBSTONED (TS 516cb07ce3064ccbbca35e912d62816c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59024
I20260805 22:24:08.550767 607 consensus_queue.cc:1035] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:08.553771 381 raft_consensus.cc:2932] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.558753 32591 raft_consensus.cc:2932] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.563304 31596 raft_consensus.cc:2932] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.568666 651 tablet_replica.cc:330] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c: stopping tablet replica
I20260805 22:24:08.569892 651 raft_consensus.cc:2229] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:08.570653 651 raft_consensus.cc:2258] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:08.574781 651 ts_tablet_manager.cc:1905] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:08.575371 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 3095cdb25afc4c26bd9d1e03a919b474 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:08.575132 31356 catalog_manager.cc:5495] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:08.585323 31351 catalog_manager.cc:5495] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:08.596791 651 ts_tablet_manager.cc:1918] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:08.597319 651 log.cc:1196] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 516cb07ce3064ccbbca35e912d62816c: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-1-root/wals/9d68a856424b4ceb8f4f54b4050e8dc4
I20260805 22:24:08.599185 31339 catalog_manager.cc:4841] TS 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): tablet 9d68a856424b4ceb8f4f54b4050e8dc4 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:08.599768 31478 tablet_service.cc:1514] Processing DeleteTablet for tablet f0eb14e6bb5c4855859507799f2b9e16 with delete_type TABLET_DATA_TOMBSTONED (TS f8e9f4c756da4203917c464e2b179994 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40714
I20260805 22:24:08.602129 652 tablet_replica.cc:330] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994: stopping tablet replica
I20260805 22:24:08.602805 31484 tablet_service.cc:1514] Processing DeleteTablet for tablet 3f76fdd524184682ae27abc293a7fff6 with delete_type TABLET_DATA_TOMBSTONED (TS f8e9f4c756da4203917c464e2b179994 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40714
I20260805 22:24:08.603021 31698 tablet_service.cc:1514] Processing DeleteTablet for tablet 3095cdb25afc4c26bd9d1e03a919b474 with delete_type TABLET_DATA_TOMBSTONED (TS feb5e7ab4b1541beaf2674c3c5500385 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56246
I20260805 22:24:08.609223 653 tablet_replica.cc:330] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385: stopping tablet replica
I20260805 22:24:08.610163 652 raft_consensus.cc:2229] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:08.610471 653 raft_consensus.cc:2229] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:08.611238 653 raft_consensus.cc:2258] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:08.612380 652 raft_consensus.cc:2258] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:08.618925 653 ts_tablet_manager.cc:1905] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:08.622051 652 ts_tablet_manager.cc:1905] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:08.635404 653 ts_tablet_manager.cc:1918] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:08.636013 653 log.cc:1196] T 3095cdb25afc4c26bd9d1e03a919b474 P feb5e7ab4b1541beaf2674c3c5500385: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-2-root/wals/3095cdb25afc4c26bd9d1e03a919b474
I20260805 22:24:08.637931 31339 catalog_manager.cc:4841] TS feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): tablet 3095cdb25afc4c26bd9d1e03a919b474 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:08.641748 652 ts_tablet_manager.cc:1918] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:08.642135 652 log.cc:1196] T f0eb14e6bb5c4855859507799f2b9e16 P f8e9f4c756da4203917c464e2b179994: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-0-root/wals/f0eb14e6bb5c4855859507799f2b9e16
I20260805 22:24:08.643520 31339 catalog_manager.cc:4841] TS f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): tablet f0eb14e6bb5c4855859507799f2b9e16 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:08.643622 652 tablet_replica.cc:330] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994: stopping tablet replica
I20260805 22:24:08.644474 652 raft_consensus.cc:2229] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:08.645061 652 raft_consensus.cc:2258] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:08.647929 652 ts_tablet_manager.cc:1905] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:08.658494 652 ts_tablet_manager.cc:1918] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:08.658795 652 log.cc:1196] T 3f76fdd524184682ae27abc293a7fff6 P f8e9f4c756da4203917c464e2b179994: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-0-root/wals/3f76fdd524184682ae27abc293a7fff6
I20260805 22:24:08.659931 31339 catalog_manager.cc:4841] TS f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): tablet 3f76fdd524184682ae27abc293a7fff6 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:09.656078 31602 consensus_queue.cc:237] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:09.663070 31709 raft_consensus.cc:1261] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260805 22:24:09.663713 31556 consensus_peers.cc:487] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c43b2a80b8c446bdb2ac0e9ca04e8cf4. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:09.664167 632 consensus_queue.cc:1035] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:09.664793 31492 raft_consensus.cc:1261] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:09.666663 632 consensus_queue.cc:1035] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:09.671759 429 raft_consensus.cc:2932] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:09.673545 31709 raft_consensus.cc:2932] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:09.676266 31492 raft_consensus.cc:2932] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:09.696945 31494 consensus_queue.cc:237] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:09.710181 31359 catalog_manager.cc:5495] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260805 22:24:09.725185 31447 consensus_peers.cc:487] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f914c8243e724eda849ce1fc2dc30827. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:09.727082 31602 raft_consensus.cc:1261] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:09.729954 381 consensus_queue.cc:1035] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:09.727375 31709 raft_consensus.cc:1261] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:09.732806 607 consensus_queue.cc:1035] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:09.740693 607 raft_consensus.cc:2932] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:09.743934 31594 raft_consensus.cc:2932] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:09.748150 31706 raft_consensus.cc:2932] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:09.758117 31494 consensus_queue.cc:237] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:09.764969 31709 raft_consensus.cc:1261] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260805 22:24:09.765156 31447 consensus_peers.cc:487] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d264e8379ab54b4785c836a00397ba95. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:09.765836 31594 raft_consensus.cc:1261] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:09.766366 381 consensus_queue.cc:1035] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:09.767159 607 consensus_queue.cc:1035] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:09.781059 381 raft_consensus.cc:2932] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:09.781462 31356 catalog_manager.cc:5495] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:09.783036 31602 raft_consensus.cc:2932] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:09.801201 31709 raft_consensus.cc:2932] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:09.802815 31711 consensus_queue.cc:237] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:09.806067 31356 catalog_manager.cc:5495] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:09.817343 31602 raft_consensus.cc:1261] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:09.819386 667 consensus_queue.cc:1035] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260805 22:24:09.822856 31668 consensus_peers.cc:487] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7c99e6a491454b5ba8d31e3246e77797. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:09.823966 31494 raft_consensus.cc:1261] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:09.825335 667 consensus_queue.cc:1035] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:09.838558 646 raft_consensus.cc:2932] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:09.841975 31602 raft_consensus.cc:2932] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:09.863797 31356 catalog_manager.cc:5495] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:09.869813 31494 raft_consensus.cc:2932] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:10.158957 688 ts_tablet_manager.cc:927] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:24:10.162446 688 tablet_copy_client.cc:316] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:45607
I20260805 22:24:10.164132 31516 tablet_copy_service.cc:140] P f8e9f4c756da4203917c464e2b179994: Received BeginTabletCopySession request for tablet d264e8379ab54b4785c836a00397ba95 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:52098)
I20260805 22:24:10.164695 31516 tablet_copy_service.cc:161] P f8e9f4c756da4203917c464e2b179994: Beginning new tablet copy session on tablet d264e8379ab54b4785c836a00397ba95 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:52098: session id = 6bf8c694a0ba40978195ae044a917caf-d264e8379ab54b4785c836a00397ba95
I20260805 22:24:10.168490 692 ts_tablet_manager.cc:927] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:24:10.178171 31516 tablet_copy_source_session.cc:205] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:10.180459 692 tablet_copy_client.cc:316] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:45607
I20260805 22:24:10.181586 688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d264e8379ab54b4785c836a00397ba95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:10.182494 31516 tablet_copy_service.cc:140] P f8e9f4c756da4203917c464e2b179994: Received BeginTabletCopySession request for tablet f914c8243e724eda849ce1fc2dc30827 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:52098)
I20260805 22:24:10.183493 31516 tablet_copy_service.cc:161] P f8e9f4c756da4203917c464e2b179994: Beginning new tablet copy session on tablet f914c8243e724eda849ce1fc2dc30827 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:52098: session id = 6bf8c694a0ba40978195ae044a917caf-f914c8243e724eda849ce1fc2dc30827
I20260805 22:24:10.192214 31516 tablet_copy_source_session.cc:205] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:10.195631 692 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f914c8243e724eda849ce1fc2dc30827. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:10.200397 688 tablet_copy_client.cc:791] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:10.201035 688 tablet_copy_client.cc:655] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:10.204507 688 tablet_copy_client.cc:531] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:10.221630 688 tablet_bootstrap.cc:492] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:10.229866 692 tablet_copy_client.cc:791] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:10.230636 692 tablet_copy_client.cc:655] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:10.231227 694 ts_tablet_manager.cc:927] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:10.238117 692 tablet_copy_client.cc:531] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:10.251542 694 tablet_copy_client.cc:316] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:41897
I20260805 22:24:10.253904 31623 tablet_copy_service.cc:140] P 516cb07ce3064ccbbca35e912d62816c: Received BeginTabletCopySession request for tablet c43b2a80b8c446bdb2ac0e9ca04e8cf4 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:56384)
I20260805 22:24:10.254185 692 tablet_bootstrap.cc:492] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:10.254544 31623 tablet_copy_service.cc:161] P 516cb07ce3064ccbbca35e912d62816c: Beginning new tablet copy session on tablet c43b2a80b8c446bdb2ac0e9ca04e8cf4 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:56384: session id = 6bf8c694a0ba40978195ae044a917caf-c43b2a80b8c446bdb2ac0e9ca04e8cf4
I20260805 22:24:10.262250 31623 tablet_copy_source_session.cc:205] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:10.263612 688 tablet_bootstrap.cc:492] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:10.264629 688 tablet_bootstrap.cc:492] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:10.265241 688 ts_tablet_manager.cc:1397] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.044s user 0.014s sys 0.013s
I20260805 22:24:10.266625 694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c43b2a80b8c446bdb2ac0e9ca04e8cf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:10.268060 688 raft_consensus.cc:348] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:10.268908 688 raft_consensus.cc:729] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:10.269563 688 consensus_queue.cc:260] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:10.282833 692 tablet_bootstrap.cc:492] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:10.284425 692 tablet_bootstrap.cc:492] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:10.285130 692 ts_tablet_manager.cc:1397] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.011s
I20260805 22:24:10.287829 692 raft_consensus.cc:348] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:10.289634 692 raft_consensus.cc:729] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:10.290354 692 consensus_queue.cc:260] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:10.300431 692 ts_tablet_manager.cc:1428] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.015s user 0.006s sys 0.007s
I20260805 22:24:10.302542 31516 tablet_copy_service.cc:342] P f8e9f4c756da4203917c464e2b179994: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-f914c8243e724eda849ce1fc2dc30827 received from {username='slave'} at 127.0.0.1:52098
I20260805 22:24:10.303020 31516 tablet_copy_service.cc:433] P f8e9f4c756da4203917c464e2b179994: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-f914c8243e724eda849ce1fc2dc30827 on tablet f914c8243e724eda849ce1fc2dc30827 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:10.316310 688 ts_tablet_manager.cc:1428] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.051s user 0.032s sys 0.011s
I20260805 22:24:10.315544 694 tablet_copy_client.cc:791] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:10.317304 694 tablet_copy_client.cc:655] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:10.323879 31516 tablet_copy_service.cc:342] P f8e9f4c756da4203917c464e2b179994: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-d264e8379ab54b4785c836a00397ba95 received from {username='slave'} at 127.0.0.1:52098
I20260805 22:24:10.324357 31516 tablet_copy_service.cc:433] P f8e9f4c756da4203917c464e2b179994: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-d264e8379ab54b4785c836a00397ba95 on tablet d264e8379ab54b4785c836a00397ba95 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:10.326514 694 tablet_copy_client.cc:531] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:10.334105 694 tablet_bootstrap.cc:492] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:10.344871 688 ts_tablet_manager.cc:927] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:10.346729 688 tablet_copy_client.cc:316] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:44831
I20260805 22:24:10.367683 31729 tablet_copy_service.cc:140] P feb5e7ab4b1541beaf2674c3c5500385: Received BeginTabletCopySession request for tablet 7c99e6a491454b5ba8d31e3246e77797 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:58348)
I20260805 22:24:10.368304 31729 tablet_copy_service.cc:161] P feb5e7ab4b1541beaf2674c3c5500385: Beginning new tablet copy session on tablet 7c99e6a491454b5ba8d31e3246e77797 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:58348: session id = 6bf8c694a0ba40978195ae044a917caf-7c99e6a491454b5ba8d31e3246e77797
I20260805 22:24:10.386210 694 tablet_bootstrap.cc:492] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:10.387161 694 tablet_bootstrap.cc:492] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:10.387792 694 ts_tablet_manager.cc:1397] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.054s user 0.025s sys 0.025s
I20260805 22:24:10.390694 694 raft_consensus.cc:348] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:10.391696 694 raft_consensus.cc:729] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:10.392390 694 consensus_queue.cc:260] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:10.403193 694 ts_tablet_manager.cc:1428] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260805 22:24:10.392216 31729 tablet_copy_source_session.cc:205] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:10.412048 31623 tablet_copy_service.cc:342] P 516cb07ce3064ccbbca35e912d62816c: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-c43b2a80b8c446bdb2ac0e9ca04e8cf4 received from {username='slave'} at 127.0.0.1:56384
I20260805 22:24:10.412230 688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7c99e6a491454b5ba8d31e3246e77797. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:10.412469 31623 tablet_copy_service.cc:433] P 516cb07ce3064ccbbca35e912d62816c: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-c43b2a80b8c446bdb2ac0e9ca04e8cf4 on tablet c43b2a80b8c446bdb2ac0e9ca04e8cf4 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:10.426425 688 tablet_copy_client.cc:791] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:10.427181 688 tablet_copy_client.cc:655] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:10.430661 688 tablet_copy_client.cc:531] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:10.438657 688 tablet_bootstrap.cc:492] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:10.459226 688 tablet_bootstrap.cc:492] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:10.460016 688 tablet_bootstrap.cc:492] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:10.460719 688 ts_tablet_manager.cc:1397] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.010s
I20260805 22:24:10.463383 688 raft_consensus.cc:348] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:10.464241 688 raft_consensus.cc:729] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:10.464859 688 consensus_queue.cc:260] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:10.469275 688 ts_tablet_manager.cc:1428] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.008s user 0.002s sys 0.003s
I20260805 22:24:10.470933 31729 tablet_copy_service.cc:342] P feb5e7ab4b1541beaf2674c3c5500385: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-7c99e6a491454b5ba8d31e3246e77797 received from {username='slave'} at 127.0.0.1:58348
I20260805 22:24:10.471334 31729 tablet_copy_service.cc:433] P feb5e7ab4b1541beaf2674c3c5500385: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-7c99e6a491454b5ba8d31e3246e77797 on tablet 7c99e6a491454b5ba8d31e3246e77797 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:10.599882 32589 raft_consensus.cc:1203] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:10.613513 32589 raft_consensus.cc:1203] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:10.676229 32589 raft_consensus.cc:1203] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:10.834692 32589 raft_consensus.cc:1203] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:11.042776 614 raft_consensus.cc:1050] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:11.044960 614 consensus_queue.cc:237] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:11.052647 31711 raft_consensus.cc:1261] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:11.052914 32589 raft_consensus.cc:1261] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:11.055102 381 consensus_queue.cc:1035] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:11.058859 381 consensus_queue.cc:1035] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:11.061955 31594 raft_consensus.cc:1261] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:11.063426 607 consensus_queue.cc:1035] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:11.071923 381 raft_consensus.cc:2932] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.073175 31709 raft_consensus.cc:2932] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.096249 31356 catalog_manager.cc:5495] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.096653 32589 raft_consensus.cc:2932] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.098433 31594 raft_consensus.cc:2932] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.130465 607 raft_consensus.cc:1050] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:11.132537 607 consensus_queue.cc:237] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:11.138701 31706 raft_consensus.cc:1261] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:11.139750 31594 raft_consensus.cc:1261] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:11.140007 607 consensus_queue.cc:1035] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:11.140982 32589 raft_consensus.cc:1261] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:11.141258 705 consensus_queue.cc:1035] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:11.143132 705 consensus_queue.cc:1035] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:11.156562 31494 consensus_queue.cc:237] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:11.162966 32591 raft_consensus.cc:1261] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:11.164590 31604 raft_consensus.cc:1261] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:11.165653 381 consensus_queue.cc:1035] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:11.166797 607 consensus_queue.cc:1035] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:11.175429 705 raft_consensus.cc:2932] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.182297 715 raft_consensus.cc:2932] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.184240 32591 raft_consensus.cc:2932] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.191332 660 raft_consensus.cc:1050] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:11.194243 32589 raft_consensus.cc:2932] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.200372 660 consensus_queue.cc:237] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:11.205358 31359 catalog_manager.cc:5495] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.209980 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet d264e8379ab54b4785c836a00397ba95 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:11.211521 31594 raft_consensus.cc:2932] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.219851 31711 raft_consensus.cc:2932] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.224903 31357 catalog_manager.cc:5495] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.238240 31698 tablet_service.cc:1514] Processing DeleteTablet for tablet d264e8379ab54b4785c836a00397ba95 with delete_type TABLET_DATA_TOMBSTONED (TS feb5e7ab4b1541beaf2674c3c5500385 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56246
I20260805 22:24:11.240741 726 tablet_replica.cc:330] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385: stopping tablet replica
I20260805 22:24:11.241956 726 raft_consensus.cc:2229] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:11.242882 726 raft_consensus.cc:2258] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:11.246518 726 ts_tablet_manager.cc:1905] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:11.251044 31492 consensus_queue.cc:237] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:11.259357 646 raft_consensus.cc:1050] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:11.264590 31711 raft_consensus.cc:1261] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:11.264736 32589 raft_consensus.cc:1261] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:11.265697 646 consensus_queue.cc:237] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:11.266851 31596 raft_consensus.cc:2932] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.268229 715 consensus_queue.cc:1035] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:24:11.282284 32589 raft_consensus.cc:2932] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.285104 614 consensus_queue.cc:1035] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:11.287128 381 raft_consensus.cc:2932] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.298286 31711 raft_consensus.cc:2932] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.309851 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet f914c8243e724eda849ce1fc2dc30827 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:11.310185 31357 catalog_manager.cc:5495] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.324023 726 ts_tablet_manager.cc:1918] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:11.324661 726 log.cc:1196] T d264e8379ab54b4785c836a00397ba95 P feb5e7ab4b1541beaf2674c3c5500385: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-2-root/wals/d264e8379ab54b4785c836a00397ba95
I20260805 22:24:11.327368 31339 catalog_manager.cc:4841] TS feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): tablet d264e8379ab54b4785c836a00397ba95 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:11.375658 31581 tablet_service.cc:1514] Processing DeleteTablet for tablet f914c8243e724eda849ce1fc2dc30827 with delete_type TABLET_DATA_TOMBSTONED (TS 516cb07ce3064ccbbca35e912d62816c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59024
I20260805 22:24:11.385547 731 tablet_replica.cc:330] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c: stopping tablet replica
I20260805 22:24:11.393168 731 raft_consensus.cc:2229] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:11.394332 731 raft_consensus.cc:2258] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:11.398722 731 ts_tablet_manager.cc:1905] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:11.403412 32589 raft_consensus.cc:1261] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:11.404816 31494 raft_consensus.cc:1261] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:11.405453 32589 raft_consensus.cc:1261] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:11.406415 31492 raft_consensus.cc:1261] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:11.407428 646 consensus_queue.cc:1035] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:11.408231 662 consensus_queue.cc:1035] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:11.408623 687 consensus_queue.cc:1035] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:11.408325 31711 raft_consensus.cc:1261] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:11.413244 31604 raft_consensus.cc:1261] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:11.416393 662 consensus_queue.cc:1035] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.009s
I20260805 22:24:11.421241 728 consensus_queue.cc:1035] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260805 22:24:11.429590 646 consensus_queue.cc:1035] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.016s
I20260805 22:24:11.451114 686 raft_consensus.cc:2932] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.455377 31494 raft_consensus.cc:2932] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.458287 662 raft_consensus.cc:2932] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.465126 32589 raft_consensus.cc:2932] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.468567 31594 raft_consensus.cc:2932] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.469831 32587 raft_consensus.cc:2932] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.490805 31351 catalog_manager.cc:5495] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:11.501248 31502 raft_consensus.cc:2932] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.503908 31357 catalog_manager.cc:5495] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.494143 31709 raft_consensus.cc:2932] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.526147 731 ts_tablet_manager.cc:1918] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:11.526680 731 log.cc:1196] T f914c8243e724eda849ce1fc2dc30827 P 516cb07ce3064ccbbca35e912d62816c: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-1-root/wals/f914c8243e724eda849ce1fc2dc30827
I20260805 22:24:11.526938 31604 consensus_queue.cc:237] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:11.534219 31339 catalog_manager.cc:4841] TS 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): tablet f914c8243e724eda849ce1fc2dc30827 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:11.542596 31502 raft_consensus.cc:1261] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:11.545243 632 consensus_queue.cc:1035] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:11.557466 32589 raft_consensus.cc:1261] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:11.559468 632 consensus_queue.cc:1035] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:11.567128 660 raft_consensus.cc:2932] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.568012 31502 raft_consensus.cc:2932] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.577804 32589 raft_consensus.cc:2932] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.580804 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c43b2a80b8c446bdb2ac0e9ca04e8cf4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:11.589145 31357 catalog_manager.cc:5495] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New cstate: current_term: 1 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.603916 31698 tablet_service.cc:1514] Processing DeleteTablet for tablet c43b2a80b8c446bdb2ac0e9ca04e8cf4 with delete_type TABLET_DATA_TOMBSTONED (TS feb5e7ab4b1541beaf2674c3c5500385 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56246
I20260805 22:24:11.604846 726 tablet_replica.cc:330] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385: stopping tablet replica
I20260805 22:24:11.605976 726 raft_consensus.cc:2229] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:11.606748 726 raft_consensus.cc:2258] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:11.609951 726 ts_tablet_manager.cc:1905] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:11.626626 726 ts_tablet_manager.cc:1918] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:11.627115 726 log.cc:1196] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P feb5e7ab4b1541beaf2674c3c5500385: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-2-root/wals/c43b2a80b8c446bdb2ac0e9ca04e8cf4
I20260805 22:24:11.628418 31339 catalog_manager.cc:4841] TS feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): tablet c43b2a80b8c446bdb2ac0e9ca04e8cf4 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:11.942955 31706 consensus_queue.cc:237] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:11.946849 32587 raft_consensus.cc:1261] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:11.948645 646 consensus_queue.cc:1035] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:11.950639 31604 raft_consensus.cc:1261] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:11.951848 646 consensus_queue.cc:1035] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:11.956722 686 raft_consensus.cc:2932] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.958215 32587 raft_consensus.cc:2932] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.960897 31604 raft_consensus.cc:2932] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.967694 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 7c99e6a491454b5ba8d31e3246e77797 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:11.980038 31351 catalog_manager.cc:5495] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:11.987756 31484 tablet_service.cc:1514] Processing DeleteTablet for tablet 7c99e6a491454b5ba8d31e3246e77797 with delete_type TABLET_DATA_TOMBSTONED (TS f8e9f4c756da4203917c464e2b179994 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40714
I20260805 22:24:11.989734 745 tablet_replica.cc:330] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994: stopping tablet replica
I20260805 22:24:11.990634 745 raft_consensus.cc:2229] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:11.991245 745 raft_consensus.cc:2258] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:11.993963 745 ts_tablet_manager.cc:1905] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:12.006451 745 ts_tablet_manager.cc:1918] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:12.006763 745 log.cc:1196] T 7c99e6a491454b5ba8d31e3246e77797 P f8e9f4c756da4203917c464e2b179994: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-0-root/wals/7c99e6a491454b5ba8d31e3246e77797
I20260805 22:24:12.008001 31339 catalog_manager.cc:4841] TS f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): tablet 7c99e6a491454b5ba8d31e3246e77797 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:13.043020 31502 consensus_queue.cc:237] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:13.049853 31706 raft_consensus.cc:1261] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260805 22:24:13.050133 31447 consensus_peers.cc:487] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 986c28960d44411487f79ca134af59e2. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:13.050978 381 consensus_queue.cc:1035] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:13.053107 31604 raft_consensus.cc:1261] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:13.054209 381 consensus_queue.cc:1035] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:13.057571 751 raft_consensus.cc:2932] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:13.059115 31706 raft_consensus.cc:2932] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:13.064492 31604 raft_consensus.cc:2932] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:13.076027 31502 consensus_queue.cc:237] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:13.082338 31356 catalog_manager.cc:5495] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
W20260805 22:24:13.092109 31447 consensus_peers.cc:487] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc38ff59fd654f6689e3a88fdbb765cf. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:13.096096 31706 raft_consensus.cc:1261] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:13.096114 31604 raft_consensus.cc:1261] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:13.098439 381 consensus_queue.cc:1035] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260805 22:24:13.100943 751 consensus_queue.cc:1035] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:13.145005 31706 raft_consensus.cc:2932] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:13.142735 381 raft_consensus.cc:2932] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:13.149562 31604 raft_consensus.cc:2932] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:13.174163 31706 consensus_queue.cc:237] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:13.180404 31357 catalog_manager.cc:5495] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:13.193497 31604 raft_consensus.cc:1261] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:13.194895 686 consensus_queue.cc:1035] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:13.198832 31502 raft_consensus.cc:1261] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:13.199959 686 consensus_queue.cc:1035] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20260805 22:24:13.204101 31668 consensus_peers.cc:487] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 16e20d6ca2f34a65a5d7bc4471282feb. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:13.219595 686 raft_consensus.cc:2932] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:13.222965 31502 raft_consensus.cc:2932] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:13.223407 31594 raft_consensus.cc:2932] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:13.234524 31711 consensus_queue.cc:237] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:13.244730 31357 catalog_manager.cc:5495] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:13.249343 31594 raft_consensus.cc:1261] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260805 22:24:13.250524 31668 consensus_peers.cc:487] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4172338662524cb29310f48e0861873e. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:13.250619 31502 raft_consensus.cc:1261] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:13.251385 775 consensus_queue.cc:1035] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:13.252290 686 consensus_queue.cc:1035] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:13.259995 775 raft_consensus.cc:2932] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:13.262799 31594 raft_consensus.cc:2932] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:13.280217 31492 raft_consensus.cc:2932] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:13.290642 31356 catalog_manager.cc:5495] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:13.531896 784 ts_tablet_manager.cc:927] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:24:13.533788 784 tablet_copy_client.cc:316] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:45607
I20260805 22:24:13.535254 31516 tablet_copy_service.cc:140] P f8e9f4c756da4203917c464e2b179994: Received BeginTabletCopySession request for tablet 986c28960d44411487f79ca134af59e2 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:52098)
I20260805 22:24:13.535758 31516 tablet_copy_service.cc:161] P f8e9f4c756da4203917c464e2b179994: Beginning new tablet copy session on tablet 986c28960d44411487f79ca134af59e2 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:52098: session id = 6bf8c694a0ba40978195ae044a917caf-986c28960d44411487f79ca134af59e2
I20260805 22:24:13.541576 31516 tablet_copy_source_session.cc:205] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:13.544013 784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 986c28960d44411487f79ca134af59e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:13.559799 784 tablet_copy_client.cc:791] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:13.560362 784 tablet_copy_client.cc:655] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:13.565506 784 tablet_copy_client.cc:531] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:13.573423 784 tablet_bootstrap.cc:492] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:13.592701 788 ts_tablet_manager.cc:927] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:24:13.593084 784 tablet_bootstrap.cc:492] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:13.594058 784 tablet_bootstrap.cc:492] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:13.594853 784 ts_tablet_manager.cc:1397] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.022s user 0.019s sys 0.004s
I20260805 22:24:13.596367 788 tablet_copy_client.cc:316] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:45607
I20260805 22:24:13.597721 31516 tablet_copy_service.cc:140] P f8e9f4c756da4203917c464e2b179994: Received BeginTabletCopySession request for tablet dc38ff59fd654f6689e3a88fdbb765cf from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:52098)
I20260805 22:24:13.597602 784 raft_consensus.cc:348] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:13.598245 31516 tablet_copy_service.cc:161] P f8e9f4c756da4203917c464e2b179994: Beginning new tablet copy session on tablet dc38ff59fd654f6689e3a88fdbb765cf from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:52098: session id = 6bf8c694a0ba40978195ae044a917caf-dc38ff59fd654f6689e3a88fdbb765cf
I20260805 22:24:13.598433 784 raft_consensus.cc:729] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:13.599086 784 consensus_queue.cc:260] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:13.609055 31516 tablet_copy_source_session.cc:205] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:13.613808 784 ts_tablet_manager.cc:1428] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.019s user 0.005s sys 0.011s
I20260805 22:24:13.615715 788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dc38ff59fd654f6689e3a88fdbb765cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:13.617262 31516 tablet_copy_service.cc:342] P f8e9f4c756da4203917c464e2b179994: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-986c28960d44411487f79ca134af59e2 received from {username='slave'} at 127.0.0.1:52098
I20260805 22:24:13.617779 31516 tablet_copy_service.cc:433] P f8e9f4c756da4203917c464e2b179994: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-986c28960d44411487f79ca134af59e2 on tablet 986c28960d44411487f79ca134af59e2 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:13.631701 788 tablet_copy_client.cc:791] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:13.632340 788 tablet_copy_client.cc:655] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:13.635833 788 tablet_copy_client.cc:531] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:13.643635 788 tablet_bootstrap.cc:492] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:13.663810 788 tablet_bootstrap.cc:492] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:13.664659 788 tablet_bootstrap.cc:492] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:13.665330 788 ts_tablet_manager.cc:1397] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.022s user 0.012s sys 0.009s
I20260805 22:24:13.667539 788 raft_consensus.cc:348] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:13.668282 788 raft_consensus.cc:729] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:13.668905 788 consensus_queue.cc:260] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:13.671000 788 ts_tablet_manager.cc:1428] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260805 22:24:13.672830 31516 tablet_copy_service.cc:342] P f8e9f4c756da4203917c464e2b179994: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-dc38ff59fd654f6689e3a88fdbb765cf received from {username='slave'} at 127.0.0.1:52098
I20260805 22:24:13.673242 31516 tablet_copy_service.cc:433] P f8e9f4c756da4203917c464e2b179994: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-dc38ff59fd654f6689e3a88fdbb765cf on tablet dc38ff59fd654f6689e3a88fdbb765cf with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:13.759824 788 ts_tablet_manager.cc:927] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:13.761408 788 tablet_copy_client.cc:316] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:44831
I20260805 22:24:13.762823 31729 tablet_copy_service.cc:140] P feb5e7ab4b1541beaf2674c3c5500385: Received BeginTabletCopySession request for tablet 16e20d6ca2f34a65a5d7bc4471282feb from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:58348)
I20260805 22:24:13.763352 31729 tablet_copy_service.cc:161] P feb5e7ab4b1541beaf2674c3c5500385: Beginning new tablet copy session on tablet 16e20d6ca2f34a65a5d7bc4471282feb from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:58348: session id = 6bf8c694a0ba40978195ae044a917caf-16e20d6ca2f34a65a5d7bc4471282feb
I20260805 22:24:13.769781 31729 tablet_copy_source_session.cc:205] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:13.772431 788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16e20d6ca2f34a65a5d7bc4471282feb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:13.786654 788 tablet_copy_client.cc:791] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:13.787276 788 tablet_copy_client.cc:655] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:13.790580 788 tablet_copy_client.cc:531] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:13.800691 788 tablet_bootstrap.cc:492] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:13.815582 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:13.823925 788 tablet_bootstrap.cc:492] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:13.824715 788 tablet_bootstrap.cc:492] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:13.825344 788 ts_tablet_manager.cc:1397] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.025s user 0.013s sys 0.007s
I20260805 22:24:13.828115 788 raft_consensus.cc:348] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:13.828989 788 raft_consensus.cc:729] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:13.829672 788 consensus_queue.cc:260] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:13.832027 788 ts_tablet_manager.cc:1428] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260805 22:24:13.835085 31729 tablet_copy_service.cc:342] P feb5e7ab4b1541beaf2674c3c5500385: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-16e20d6ca2f34a65a5d7bc4471282feb received from {username='slave'} at 127.0.0.1:58348
I20260805 22:24:13.835484 31729 tablet_copy_service.cc:433] P feb5e7ab4b1541beaf2674c3c5500385: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-16e20d6ca2f34a65a5d7bc4471282feb on tablet 16e20d6ca2f34a65a5d7bc4471282feb with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:13.866945 788 ts_tablet_manager.cc:927] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:13.868484 788 tablet_copy_client.cc:316] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:44831
I20260805 22:24:13.870031 31729 tablet_copy_service.cc:140] P feb5e7ab4b1541beaf2674c3c5500385: Received BeginTabletCopySession request for tablet 4172338662524cb29310f48e0861873e from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:58348)
I20260805 22:24:13.874156 31729 tablet_copy_service.cc:161] P feb5e7ab4b1541beaf2674c3c5500385: Beginning new tablet copy session on tablet 4172338662524cb29310f48e0861873e from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:58348: session id = 6bf8c694a0ba40978195ae044a917caf-4172338662524cb29310f48e0861873e
I20260805 22:24:13.882318 31729 tablet_copy_source_session.cc:205] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:13.885951 788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4172338662524cb29310f48e0861873e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:13.891523 31596 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:43438
I20260805 22:24:13.892261 31596 raft_consensus.cc:595] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:13.896237 31596 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:43438
I20260805 22:24:13.896770 31596 raft_consensus.cc:595] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:13.905696 788 tablet_copy_client.cc:791] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:13.906370 788 tablet_copy_client.cc:655] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:13.910708 788 tablet_copy_client.cc:531] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:13.923597 788 tablet_bootstrap.cc:492] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:13.927809 31492 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:13.928809 31492 raft_consensus.cc:595] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:13.931869 31596 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:43438
I20260805 22:24:13.932338 31596 raft_consensus.cc:595] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:13.941954 31596 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
mode: GRACEFUL
new_leader_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:43438
I20260805 22:24:13.942989 31596 raft_consensus.cc:595] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Received request to transfer leadership to TS 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:13.945672 31596 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:43438
I20260805 22:24:13.948858 31596 raft_consensus.cc:595] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:13.950172 788 tablet_bootstrap.cc:492] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:13.951968 788 tablet_bootstrap.cc:492] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:13.952672 788 ts_tablet_manager.cc:1397] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.029s user 0.018s sys 0.008s
I20260805 22:24:13.953260 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260805 22:24:13.954993 788 raft_consensus.cc:348] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:13.955802 788 raft_consensus.cc:729] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:13.954949 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:24:13.956389 788 consensus_queue.cc:260] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:13.957252 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:13.958488 788 ts_tablet_manager.cc:1428] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260805 22:24:13.963716 31729 tablet_copy_service.cc:342] P feb5e7ab4b1541beaf2674c3c5500385: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-4172338662524cb29310f48e0861873e received from {username='slave'} at 127.0.0.1:58348
I20260805 22:24:13.964154 31729 tablet_copy_service.cc:433] P feb5e7ab4b1541beaf2674c3c5500385: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-4172338662524cb29310f48e0861873e on tablet 4172338662524cb29310f48e0861873e with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:14.006089 32587 raft_consensus.cc:1203] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:14.051496 777 raft_consensus.cc:979] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c: : Instructing follower feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:14.051982 777 raft_consensus.cc:1067] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Signalling peer feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:14.058465 31709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:56256
I20260805 22:24:14.059072 31709 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:14.059345 31709 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:14.069226 31709 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:14.070726 31709 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:14.071700 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:14.072342 31594 raft_consensus.cc:3032] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:14.072562 31594 raft_consensus.cc:729] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:14.073316 31594 consensus_queue.cc:260] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:14.074273 31594 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:14.078665 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:14.078864 31594 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:14.079253 31492 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:14.081298 31667 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:14.082561 775 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:14.083010 775 raft_consensus.cc:686] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:14.083797 775 consensus_queue.cc:237] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:14.085461 31492 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:14.099593 31357 catalog_manager.cc:5495] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 1 to 2, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:14.114800 632 raft_consensus.cc:979] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c: : Instructing follower 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:14.115239 777 raft_consensus.cc:1067] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Signalling peer 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:14.116629 32587 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:57372
I20260805 22:24:14.117193 32587 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:14.117571 32587 raft_consensus.cc:3037] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:14.123082 32587 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:14.125116 32587 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:14.126091 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:14.125957 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:14.126618 31711 raft_consensus.cc:3037] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:14.126765 31596 raft_consensus.cc:3032] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:14.127173 31596 raft_consensus.cc:729] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:14.127723 31596 consensus_queue.cc:260] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:14.128557 31596 raft_consensus.cc:3037] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:14.131500 31711 raft_consensus.cc:2454] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:14.132655 32529 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:14.133142 31596 raft_consensus.cc:2454] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:14.133914 790 raft_consensus.cc:2781] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:14.142037 790 raft_consensus.cc:686] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:14.143085 790 consensus_queue.cc:237] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:14.152516 31357 catalog_manager.cc:5495] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 2, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 2 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:14.209962 632 raft_consensus.cc:979] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c: : Instructing follower feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:14.210532 777 raft_consensus.cc:1067] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Signalling peer feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:14.211402 32587 raft_consensus.cc:1203] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:14.214111 31709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:56256
I20260805 22:24:14.214735 31709 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:14.215061 31709 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:14.221608 31709 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:14.223114 31709 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:14.224102 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:14.231309 31502 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:14.224102 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:14.233629 31596 raft_consensus.cc:3032] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:14.233932 31596 raft_consensus.cc:729] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:14.234606 31596 consensus_queue.cc:260] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:14.235666 31596 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:14.237990 632 raft_consensus.cc:979] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c: : Instructing follower feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:14.238454 777 raft_consensus.cc:1067] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Signalling peer feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:14.239745 31709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:56256
I20260805 22:24:14.240267 31709 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:14.240639 31709 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:14.246577 31596 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:14.247767 31667 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:14.248829 775 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:14.249089 31502 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:14.249627 775 raft_consensus.cc:686] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:14.250471 775 consensus_queue.cc:237] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:14.249402 31709 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:14.254731 31709 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:14.255626 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:14.256122 31596 raft_consensus.cc:3032] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:14.256342 31596 raft_consensus.cc:729] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:14.256913 31596 consensus_queue.cc:260] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:14.257757 31596 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:14.261922 32587 raft_consensus.cc:1203] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:14.264101 31596 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:14.258275 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:14.266273 31502 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:14.270581 31357 catalog_manager.cc:5495] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 1 to 2, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:14.273622 31502 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:14.276985 31667 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:14.278724 775 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:14.279270 775 raft_consensus.cc:686] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:14.280416 775 consensus_queue.cc:237] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:14.301349 31357 catalog_manager.cc:5495] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 1 to 2, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:14.324839 32587 raft_consensus.cc:1203] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:14.348161 814 raft_consensus.cc:979] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c: : Instructing follower feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:14.348702 814 raft_consensus.cc:1067] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Signalling peer feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:14.350114 31711 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:56256
I20260805 22:24:14.350660 31711 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:14.350910 31711 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:14.356283 31711 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:14.357752 31711 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:14.358472 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:14.358768 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:14.359097 31502 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:14.359285 31596 raft_consensus.cc:3032] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:14.359575 31596 raft_consensus.cc:729] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:14.360144 31596 consensus_queue.cc:260] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:14.360991 31596 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:14.365085 31502 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:14.366104 31667 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:14.366748 686 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:14.367254 686 raft_consensus.cc:686] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:14.368518 686 consensus_queue.cc:237] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:14.374826 31596 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:14.379346 31357 catalog_manager.cc:5495] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 1 to 2, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:14.440315 381 raft_consensus.cc:979] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994: : Instructing follower 516cb07ce3064ccbbca35e912d62816c to start an election
I20260805 22:24:14.440763 381 raft_consensus.cc:1067] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Signalling peer 516cb07ce3064ccbbca35e912d62816c to start an election
I20260805 22:24:14.442132 31596 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:59028
I20260805 22:24:14.442561 31596 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:14.442832 31596 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:14.449826 31596 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:14.451596 31596 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:14.452730 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:14.453346 31709 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:14.453599 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:14.454195 31492 raft_consensus.cc:3032] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:14.454442 31492 raft_consensus.cc:729] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:14.455701 31492 consensus_queue.cc:260] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:14.456740 31492 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:14.460610 31709 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:14.462334 31554 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:14.463053 777 raft_consensus.cc:2781] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:14.463573 777 raft_consensus.cc:686] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:14.464833 777 consensus_queue.cc:237] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:14.465773 31492 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:14.476531 31357 catalog_manager.cc:5495] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 1 to 2, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 2 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:14.530089 381 raft_consensus.cc:1050] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:14.532243 381 consensus_queue.cc:237] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:14.537060 31709 raft_consensus.cc:1261] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:14.538542 790 consensus_queue.cc:1035] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:14.560547 31596 raft_consensus.cc:1261] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:14.560716 31709 raft_consensus.cc:1261] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:14.562136 381 consensus_queue.cc:1035] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:14.563059 764 consensus_queue.cc:1035] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:14.565328 31596 raft_consensus.cc:1261] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:14.566741 811 consensus_queue.cc:1035] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:14.576429 32587 raft_consensus.cc:1261] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:14.578138 31492 raft_consensus.cc:1261] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:14.579487 686 consensus_queue.cc:1035] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:14.581259 381 consensus_queue.cc:1035] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:14.617506 764 raft_consensus.cc:2932] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:14.622337 31596 raft_consensus.cc:1261] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:14.622658 31602 raft_consensus.cc:2932] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:14.623674 818 consensus_queue.cc:1035] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:14.631619 31709 raft_consensus.cc:2932] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:14.648882 31502 raft_consensus.cc:1261] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:14.653434 821 consensus_queue.cc:1035] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:14.661707 31351 catalog_manager.cc:5495] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:14.670362 32587 raft_consensus.cc:2932] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:14.671802 762 raft_consensus.cc:1050] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:14.678845 762 consensus_queue.cc:237] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:14.695192 31602 raft_consensus.cc:1261] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:14.696337 775 consensus_queue.cc:1035] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:14.705710 32587 raft_consensus.cc:1261] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:14.707623 828 consensus_queue.cc:1035] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:14.715224 31502 consensus_queue.cc:237] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:14.750758 851 log.cc:924] Time spent T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.051s user 0.001s sys 0.000s
I20260805 22:24:14.747797 31604 raft_consensus.cc:1261] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:14.753569 821 raft_consensus.cc:1050] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:14.761389 32589 raft_consensus.cc:1261] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:14.754271 31709 raft_consensus.cc:1261] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:14.756819 31495 raft_consensus.cc:1261] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:14.762671 31596 raft_consensus.cc:1261] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:14.764020 764 consensus_queue.cc:1035] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:14.767915 31602 raft_consensus.cc:1261] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:14.768775 818 consensus_queue.cc:1035] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:14.768743 764 consensus_queue.cc:1035] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:14.769439 828 consensus_queue.cc:1035] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:14.782495 762 consensus_queue.cc:1035] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:14.783455 381 raft_consensus.cc:2932] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:14.774729 821 consensus_queue.cc:237] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:14.788300 31596 raft_consensus.cc:2932] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:14.796237 764 raft_consensus.cc:2932] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:14.800256 686 consensus_queue.cc:1035] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:14.800033 32589 raft_consensus.cc:2932] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:14.813421 818 raft_consensus.cc:1050] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:14.818881 818 consensus_queue.cc:237] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:14.821199 31601 raft_consensus.cc:1261] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:14.801467 31602 raft_consensus.cc:2932] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:14.846458 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 986c28960d44411487f79ca134af59e2 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:14.847198 775 consensus_queue.cc:1035] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:14.861809 32589 raft_consensus.cc:2932] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:14.863266 31357 catalog_manager.cc:5495] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 3 to 4, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:14.866225 31502 raft_consensus.cc:1261] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:14.873665 775 consensus_queue.cc:1035] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260805 22:24:14.880703 31698 tablet_service.cc:1514] Processing DeleteTablet for tablet 986c28960d44411487f79ca134af59e2 with delete_type TABLET_DATA_TOMBSTONED (TS feb5e7ab4b1541beaf2674c3c5500385 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56246
I20260805 22:24:14.895287 31356 catalog_manager.cc:5495] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:14.896971 31495 raft_consensus.cc:1261] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:14.903755 32587 raft_consensus.cc:1261] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:14.904170 814 consensus_queue.cc:1035] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:14.910319 31596 raft_consensus.cc:1261] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:14.911360 31494 raft_consensus.cc:1261] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:14.915189 686 consensus_queue.cc:1035] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:14.924012 31711 raft_consensus.cc:1261] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:14.934670 777 consensus_queue.cc:1035] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:14.935992 818 consensus_queue.cc:1035] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:14.938465 863 tablet_replica.cc:330] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385: stopping tablet replica
I20260805 22:24:14.940721 863 raft_consensus.cc:2229] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:14.946633 863 raft_consensus.cc:2258] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:14.951727 31494 consensus_queue.cc:237] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:14.937952 31711 raft_consensus.cc:2932] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:14.965286 686 consensus_queue.cc:1035] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:15.006645 863 ts_tablet_manager.cc:1905] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:15.017339 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260805 22:24:15.032276 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 63033.9us
W20260805 22:24:15.056334 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 86305.6us
I20260805 22:24:15.057888 32589 raft_consensus.cc:1261] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:15.064615 31709 raft_consensus.cc:1261] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:15.067152 872 consensus_queue.cc:1035] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:24:15.068152 31596 raft_consensus.cc:1261] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:15.073349 686 raft_consensus.cc:2932] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:15.086575 31502 raft_consensus.cc:1261] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:15.100589 872 consensus_queue.cc:1035] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:15.104004 863 ts_tablet_manager.cc:1918] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:15.104586 863 log.cc:1196] T 986c28960d44411487f79ca134af59e2 P feb5e7ab4b1541beaf2674c3c5500385: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-2-root/wals/986c28960d44411487f79ca134af59e2
I20260805 22:24:15.105985 883 consensus_queue.cc:1035] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:15.109477 878 consensus_queue.cc:1035] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:15.109007 31339 catalog_manager.cc:4841] TS feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): tablet 986c28960d44411487f79ca134af59e2 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:15.108601 31601 raft_consensus.cc:2932] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:15.157159 32589 raft_consensus.cc:2932] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:15.157325 31704 raft_consensus.cc:2932] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:15.163354 764 raft_consensus.cc:2932] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
W20260805 22:24:15.188282 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 50435.7us
I20260805 22:24:15.183499 31502 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:15.196650 31502 raft_consensus.cc:595] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Received request to transfer leadership to TS 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:15.204681 32591 raft_consensus.cc:1261] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:15.207077 32594 raft_consensus.cc:2932] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:15.244693 31493 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:15.245982 875 consensus_queue.cc:1035] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:15.248596 31357 catalog_manager.cc:5495] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:15.250385 31493 raft_consensus.cc:595] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:15.268144 31493 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:15.268895 31493 raft_consensus.cc:595] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Received request to transfer leadership to TS 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:15.281927 31581 tablet_service.cc:1514] Processing DeleteTablet for tablet dc38ff59fd654f6689e3a88fdbb765cf with delete_type TABLET_DATA_TOMBSTONED (TS 516cb07ce3064ccbbca35e912d62816c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59024
I20260805 22:24:15.282166 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet dc38ff59fd654f6689e3a88fdbb765cf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:15.281369 31356 catalog_manager.cc:5495] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:15.281064 31601 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
mode: GRACEFUL
new_leader_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:43438
I20260805 22:24:15.315672 875 raft_consensus.cc:2932] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:15.323444 31601 raft_consensus.cc:595] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Received request to transfer leadership to TS 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:15.337626 31596 raft_consensus.cc:2932] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
W20260805 22:24:15.343710 885 log.cc:924] Time spent T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20260805 22:24:15.355650 31711 consensus_queue.cc:237] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:15.379372 31351 catalog_manager.cc:5495] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:15.407130 31596 raft_consensus.cc:1261] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:15.413836 875 consensus_queue.cc:1035] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:15.412972 31495 raft_consensus.cc:2932] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:15.415522 32594 raft_consensus.cc:1261] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:15.417130 875 consensus_queue.cc:1035] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:24:15.430784 872 raft_consensus.cc:979] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994: : Instructing follower 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:15.431591 764 raft_consensus.cc:1067] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Signalling peer 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:15.435086 883 raft_consensus.cc:2932] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:15.438937 32594 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:57382
I20260805 22:24:15.439520 32594 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:15.440078 32594 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:15.432763 893 tablet_replica.cc:330] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c: stopping tablet replica
I20260805 22:24:15.445940 32594 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:15.452275 872 raft_consensus.cc:979] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994: : Instructing follower 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:15.452862 762 raft_consensus.cc:1067] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Signalling peer 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:15.454222 32589 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:57382
I20260805 22:24:15.454726 32589 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:15.455036 32589 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:15.456441 31596 raft_consensus.cc:2932] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:15.462296 32589 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:15.461934 32587 raft_consensus.cc:2932] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:15.470588 32589 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:15.471279 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:15.471961 31495 raft_consensus.cc:3032] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:15.472332 31495 raft_consensus.cc:729] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:15.473075 893 raft_consensus.cc:2229] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:15.473883 893 raft_consensus.cc:2258] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:15.474267 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 16e20d6ca2f34a65a5d7bc4471282feb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:15.477531 893 ts_tablet_manager.cc:1905] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:15.477871 31495 consensus_queue.cc:260] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:15.479573 31495 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:15.482146 31711 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:43958
I20260805 22:24:15.482734 31711 raft_consensus.cc:595] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:15.482839 32587 raft_consensus.cc:2932] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:15.485076 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:15.489041 31711 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:15.498015 31492 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fe370453251f4300a8d90c397021595c"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:15.498618 31492 raft_consensus.cc:595] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Received request to transfer leadership to TS 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:15.499461 31495 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:15.499938 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260805 22:24:15.501258 762 raft_consensus.cc:979] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994: : Instructing follower 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:15.501956 882 raft_consensus.cc:1067] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Signalling peer 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:15.502636 893 ts_tablet_manager.cc:1918] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:15.503096 893 log.cc:1196] T dc38ff59fd654f6689e3a88fdbb765cf P 516cb07ce3064ccbbca35e912d62816c: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-1-root/wals/dc38ff59fd654f6689e3a88fdbb765cf
I20260805 22:24:15.503595 32529 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:15.504451 32589 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fe370453251f4300a8d90c397021595c"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:57382
I20260805 22:24:15.504974 32589 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:15.505293 32589 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:15.507954 790 raft_consensus.cc:2781] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:15.508288 32594 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:15.510304 31711 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:15.511787 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:24:15.512416 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:15.512351 31356 catalog_manager.cc:5495] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c reported cstate change: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:15.515517 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:15.516062 31492 raft_consensus.cc:3032] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:15.516373 31492 raft_consensus.cc:729] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:15.505820 31339 catalog_manager.cc:4841] TS 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): tablet dc38ff59fd654f6689e3a88fdbb765cf (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:15.517045 31492 consensus_queue.cc:260] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:15.518285 31492 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:15.513012 32589 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:15.515946 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:15.522054 31596 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:15.522740 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:15.523353 31601 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:15.525040 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:15.525650 31493 raft_consensus.cc:3032] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:15.525929 31493 raft_consensus.cc:729] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:15.526665 31493 consensus_queue.cc:260] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:15.527812 31493 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:15.529021 31492 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:15.531868 31484 tablet_service.cc:1514] Processing DeleteTablet for tablet 16e20d6ca2f34a65a5d7bc4471282feb with delete_type TABLET_DATA_TOMBSTONED (TS f8e9f4c756da4203917c464e2b179994 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40714
I20260805 22:24:15.533685 32529 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:15.534591 811 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:15.535075 811 raft_consensus.cc:686] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:15.535945 811 consensus_queue.cc:237] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:15.533666 31596 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:15.538465 31601 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:15.540213 32529 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:15.542623 31493 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:15.543787 32589 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:15.547725 882 raft_consensus.cc:979] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994: : Instructing follower 516cb07ce3064ccbbca35e912d62816c to start an election
I20260805 22:24:15.548187 882 raft_consensus.cc:1067] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Signalling peer 516cb07ce3064ccbbca35e912d62816c to start an election
I20260805 22:24:15.549988 31601 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:59028
I20260805 22:24:15.550623 31601 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:15.551023 31601 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:15.569988 31709 consensus_queue.cc:237] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:15.571198 903 tablet_replica.cc:330] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994: stopping tablet replica
I20260805 22:24:15.563292 31601 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:15.570405 811 raft_consensus.cc:2781] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:15.572538 903 raft_consensus.cc:2229] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:15.573235 811 raft_consensus.cc:686] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:15.573630 903 pending_rounds.cc:62] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994: Trying to abort 1 pending ops.
I20260805 22:24:15.573904 903 pending_rounds.cc:68] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7315327610120368128 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } attrs { replace: true } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } } }
I20260805 22:24:15.576371 32589 raft_consensus.cc:1261] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:15.574196 811 consensus_queue.cc:237] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:15.577819 903 raft_consensus.cc:2894] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20260805 22:24:15.578272 903 raft_consensus.cc:2866] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20260805 22:24:15.578389 31596 raft_consensus.cc:1261] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:15.582562 790 raft_consensus.cc:686] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:15.580493 878 consensus_queue.cc:1035] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
I20260805 22:24:15.581265 903 raft_consensus.cc:2258] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:15.584762 883 consensus_queue.cc:1035] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:15.590831 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:15.591465 31709 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:15.592048 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:15.593305 31492 raft_consensus.cc:3032] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:15.593665 31492 raft_consensus.cc:729] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:15.594470 31492 consensus_queue.cc:260] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:15.596298 31492 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:15.601509 31709 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:15.604504 31554 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:15.586509 790 consensus_queue.cc:237] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:15.588804 31601 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:15.608726 32589 raft_consensus.cc:2932] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:15.620929 31492 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:15.609865 890 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:15.632378 903 ts_tablet_manager.cc:1905] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:15.625581 881 raft_consensus.cc:2932] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
W20260805 22:24:15.634321 907 log.cc:924] Time spent T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20260805 22:24:15.627700 890 raft_consensus.cc:686] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:15.650074 31596 raft_consensus.cc:2932] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:15.656067 890 consensus_queue.cc:237] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:15.663805 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet 4172338662524cb29310f48e0861873e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:15.663319 31356 catalog_manager.cc:5495] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 3 to 4, VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193) evicted. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:15.667906 31356 catalog_manager.cc:5495] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 2, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 2 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:15.672001 903 ts_tablet_manager.cc:1918] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:15.672606 903 log.cc:1196] T 16e20d6ca2f34a65a5d7bc4471282feb P f8e9f4c756da4203917c464e2b179994: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-0-root/wals/16e20d6ca2f34a65a5d7bc4471282feb
I20260805 22:24:15.676749 31339 catalog_manager.cc:4841] TS f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): tablet 16e20d6ca2f34a65a5d7bc4471282feb (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:15.681449 31484 tablet_service.cc:1514] Processing DeleteTablet for tablet 4172338662524cb29310f48e0861873e with delete_type TABLET_DATA_TOMBSTONED (TS f8e9f4c756da4203917c464e2b179994 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:40714
I20260805 22:24:15.682492 903 tablet_replica.cc:330] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994: stopping tablet replica
I20260805 22:24:15.683863 903 raft_consensus.cc:2229] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:15.684651 903 raft_consensus.cc:2258] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:15.692991 903 ts_tablet_manager.cc:1905] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:15.707901 903 ts_tablet_manager.cc:1918] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:15.708448 903 log.cc:1196] T 4172338662524cb29310f48e0861873e P f8e9f4c756da4203917c464e2b179994: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-0-root/wals/4172338662524cb29310f48e0861873e
I20260805 22:24:15.707755 31359 catalog_manager.cc:5495] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 1 to 2, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 2 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:15.706108 31356 catalog_manager.cc:5495] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 2, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 2 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:15.710879 31339 catalog_manager.cc:4841] TS f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): tablet 4172338662524cb29310f48e0861873e (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:15.710355 31356 catalog_manager.cc:5495] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 2, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 2 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:15.866178 879 raft_consensus.cc:979] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385: : Instructing follower 516cb07ce3064ccbbca35e912d62816c to start an election
I20260805 22:24:15.866674 881 raft_consensus.cc:1067] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Signalling peer 516cb07ce3064ccbbca35e912d62816c to start an election
I20260805 22:24:15.868080 31596 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:59030
I20260805 22:24:15.868587 31596 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:15.868897 31596 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:15.874564 31596 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:15.876062 31596 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:15.877000 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:15.877867 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:15.878510 31493 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:15.879500 31709 raft_consensus.cc:3032] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:15.879858 31709 raft_consensus.cc:729] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:15.880573 31709 consensus_queue.cc:260] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:15.881783 31709 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:15.885156 31493 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:15.886262 777 raft_consensus.cc:979] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c: : Instructing follower 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:15.886535 31554 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:15.886754 890 raft_consensus.cc:1067] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Signalling peer 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:15.887305 777 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:15.887773 777 raft_consensus.cc:686] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:15.888530 32589 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:57372
I20260805 22:24:15.888569 777 consensus_queue.cc:237] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:15.889236 32589 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:15.889930 32589 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:15.890703 31709 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:15.897722 32589 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:15.899784 32589 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:15.900314 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:15.900899 31596 raft_consensus.cc:3032] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:15.900815 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:15.901450 31709 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:15.901259 31596 raft_consensus.cc:729] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:15.901194 31356 catalog_manager.cc:5495] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 1 to 2, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 2 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:15.902324 31596 consensus_queue.cc:260] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:15.903659 31596 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:15.908099 31709 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:15.910029 31596 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:15.910802 32529 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:15.911494 790 raft_consensus.cc:2781] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:15.911890 790 raft_consensus.cc:686] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:15.912600 790 consensus_queue.cc:237] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:15.922305 31356 catalog_manager.cc:5495] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 2, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 2 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:15.961169 31594 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:15.962395 790 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:15.990803 31495 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:15.992259 902 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:16.014402 31594 raft_consensus.cc:1261] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:16.015678 811 consensus_queue.cc:1035] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:16.032691 31495 raft_consensus.cc:1261] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:16.034444 902 consensus_queue.cc:1035] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:16.093945 31704 raft_consensus.cc:1261] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:16.095060 902 consensus_queue.cc:1035] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:16.107162 31493 raft_consensus.cc:1261] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:16.109614 902 consensus_queue.cc:1035] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:16.213894 31493 raft_consensus.cc:1261] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:16.215029 777 consensus_queue.cc:1035] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:16.223191 31704 raft_consensus.cc:1261] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:16.225077 890 consensus_queue.cc:1035] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:16.330008 31709 raft_consensus.cc:1261] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:16.339453 902 consensus_queue.cc:1035] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260805 22:24:16.373502 31594 raft_consensus.cc:1261] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:16.375298 902 consensus_queue.cc:1035] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260805 22:24:16.442026 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 44084.6us
I20260805 22:24:16.489748 31495 raft_consensus.cc:1261] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:16.491000 890 consensus_queue.cc:1035] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:16.502213 31709 raft_consensus.cc:1261] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:16.503660 934 consensus_queue.cc:1035] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:16.606565 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:16.656647 31594 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
mode: GRACEFUL
new_leader_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:43438
I20260805 22:24:16.657533 31594 raft_consensus.cc:595] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Received request to transfer leadership to TS 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:16.659950 31495 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:16.660426 31495 raft_consensus.cc:595] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:16.662607 31594 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:43438
I20260805 22:24:16.663069 31594 raft_consensus.cc:595] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:16.665220 31495 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:16.665887 31495 raft_consensus.cc:595] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Received request to transfer leadership to TS 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:16.668673 31495 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:16.670152 632 raft_consensus.cc:979] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c: : Instructing follower 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:16.670797 777 raft_consensus.cc:1067] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Signalling peer 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:16.673767 31495 raft_consensus.cc:595] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:16.675034 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260805 22:24:16.677740 32587 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:57372
I20260805 22:24:16.678344 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:24:16.678303 32587 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:16.679095 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:16.679252 32587 raft_consensus.cc:3037] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:16.692350 32587 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:16.694481 32587 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:16.696156 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:16.697072 31704 raft_consensus.cc:3037] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:16.697863 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:16.698463 31596 raft_consensus.cc:3032] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:16.698766 31596 raft_consensus.cc:729] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:16.699575 31596 consensus_queue.cc:260] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:16.700873 31596 raft_consensus.cc:3037] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:16.704605 31704 raft_consensus.cc:2454] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:16.706329 32529 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:16.707300 902 raft_consensus.cc:2781] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:16.707777 902 raft_consensus.cc:686] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:16.711632 902 consensus_queue.cc:237] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:16.713533 31596 raft_consensus.cc:2454] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:16.734181 31357 catalog_manager.cc:5495] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 2, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 2 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:16.783308 31709 consensus_queue.cc:237] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:16.789567 31596 raft_consensus.cc:1261] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:16.789777 31493 raft_consensus.cc:1261] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260805 22:24:16.790407 31668 consensus_peers.cc:487] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6cf90ae3fc840ceb48be8c790be97d9. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:16.791157 878 consensus_queue.cc:1035] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:16.792001 938 consensus_queue.cc:1035] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:16.799757 879 raft_consensus.cc:2932] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:16.801692 31596 raft_consensus.cc:2932] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:16.802244 31495 raft_consensus.cc:2932] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:16.825579 31359 catalog_manager.cc:5495] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:16.835703 31495 consensus_queue.cc:237] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:16.847729 777 raft_consensus.cc:979] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c: : Instructing follower feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:16.848238 890 raft_consensus.cc:1067] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Signalling peer feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:16.849980 31709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:56256
I20260805 22:24:16.850621 31709 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:16.851662 31709 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:16.852288 31704 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260805 22:24:16.853009 31447 consensus_peers.cc:487] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 91a9acdbc98a4e799374fe6e8950c3a7. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:16.854616 31596 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:16.854380 943 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:16.855716 882 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:16.858568 31709 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:16.860294 31709 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:16.861891 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:16.862340 31495 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:16.867048 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:16.867666 31601 raft_consensus.cc:3032] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:16.870124 31495 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:16.867949 31601 raft_consensus.cc:729] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:16.871407 31601 consensus_queue.cc:260] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:16.872148 31667 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:16.872704 764 raft_consensus.cc:2932] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:16.873833 879 raft_consensus.cc:2781] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:16.874339 879 raft_consensus.cc:686] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:16.875217 879 consensus_queue.cc:237] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:16.875522 31704 raft_consensus.cc:2932] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:16.883266 31495 raft_consensus.cc:1226] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:16.885273 762 raft_consensus.cc:979] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994: : Instructing follower 516cb07ce3064ccbbca35e912d62816c to start an election
I20260805 22:24:16.885826 762 raft_consensus.cc:1067] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Signalling peer 516cb07ce3064ccbbca35e912d62816c to start an election
I20260805 22:24:16.888149 31602 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:59028
I20260805 22:24:16.888640 31602 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:16.888921 31602 raft_consensus.cc:3037] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:16.890286 31596 raft_consensus.cc:2932] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:16.904804 31357 catalog_manager.cc:5495] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 1 to 2, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:16.905076 31495 consensus_queue.cc:237] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:16.907480 31601 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:16.910282 31351 catalog_manager.cc:5495] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 2 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:16.924144 31602 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:16.927028 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:16.927616 31493 raft_consensus.cc:3032] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:16.927987 31493 raft_consensus.cc:729] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:16.928627 31493 consensus_queue.cc:260] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:16.929415 31601 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:16.929718 31493 raft_consensus.cc:3037] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
W20260805 22:24:16.932509 31447 consensus_peers.cc:487] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab66683a9e5e46c2b09be4afdee2a08c. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:16.927663 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:16.933462 31704 raft_consensus.cc:3037] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:16.926406 31602 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:16.935647 31709 raft_consensus.cc:1261] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:16.938633 882 consensus_queue.cc:1035] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:16.958041 31602 raft_consensus.cc:1261] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:16.959399 882 consensus_queue.cc:1035] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:16.963562 31493 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:16.965279 31554 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:16.966163 890 raft_consensus.cc:2781] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:16.966630 890 raft_consensus.cc:686] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:16.967566 890 consensus_queue.cc:237] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:16.968554 31704 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:16.975548 943 raft_consensus.cc:2932] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:16.977401 31704 raft_consensus.cc:2932] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:16.983340 31604 raft_consensus.cc:2932] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:16.994120 31351 catalog_manager.cc:5495] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 1 to 2, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 2 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:16.998802 31356 catalog_manager.cc:5495] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:17.002875 31493 consensus_queue.cc:237] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:17.015017 31709 raft_consensus.cc:1261] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:17.016633 762 consensus_queue.cc:1035] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:17.017346 31604 raft_consensus.cc:1261] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260805 22:24:17.017819 940 raft_consensus.cc:979] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994: : Instructing follower 516cb07ce3064ccbbca35e912d62816c to start an election
I20260805 22:24:17.018307 940 raft_consensus.cc:1067] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Signalling peer 516cb07ce3064ccbbca35e912d62816c to start an election
I20260805 22:24:17.021981 882 consensus_queue.cc:1035] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:17.021991 31604 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:59028
I20260805 22:24:17.024147 31604 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:17.024441 31604 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
W20260805 22:24:17.025884 31447 consensus_peers.cc:487] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c75e772a0f924080865ddf85ed1e2d86. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:24:17.033685 31604 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:17.041569 943 raft_consensus.cc:2932] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:17.043799 31709 raft_consensus.cc:2932] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:17.050364 940 raft_consensus.cc:979] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994: : Instructing follower 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:17.050868 940 raft_consensus.cc:1067] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Signalling peer 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:17.052418 32589 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:57382
I20260805 22:24:17.052961 32589 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:17.052251 31602 raft_consensus.cc:2932] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:17.053308 32589 raft_consensus.cc:3037] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:17.055809 31604 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:17.058816 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:17.059405 31493 raft_consensus.cc:3032] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:17.061910 31493 raft_consensus.cc:729] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:17.057658 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:17.064047 31704 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:17.062657 31493 consensus_queue.cc:260] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:17.065423 31493 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:17.074111 31704 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:17.081176 32589 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:17.083827 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:17.084372 31492 raft_consensus.cc:3032] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:17.084643 31492 raft_consensus.cc:729] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:17.084674 31493 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:17.085366 31492 consensus_queue.cc:260] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:17.086486 31492 raft_consensus.cc:3037] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:17.087291 31554 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:17.086812 31359 catalog_manager.cc:5495] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index -1 to 2, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:17.087990 934 raft_consensus.cc:2781] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:17.088452 934 raft_consensus.cc:686] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:17.089315 934 consensus_queue.cc:237] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:17.116705 32589 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:17.118273 31492 raft_consensus.cc:2454] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:17.119069 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:17.119652 31709 raft_consensus.cc:3037] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:17.120258 32529 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:17.121130 902 raft_consensus.cc:2781] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:17.121623 902 raft_consensus.cc:686] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:17.122440 902 consensus_queue.cc:237] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:17.131922 31709 raft_consensus.cc:2454] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:17.143872 31351 catalog_manager.cc:5495] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 1 to 2, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 2 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:17.152452 31357 catalog_manager.cc:5495] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 2, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 2 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:17.202522 31709 raft_consensus.cc:1261] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:17.203557 902 consensus_queue.cc:1035] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:17.219734 31602 raft_consensus.cc:1261] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:17.223897 790 consensus_queue.cc:1035] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:17.255832 31604 raft_consensus.cc:1261] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:17.257218 881 consensus_queue.cc:1035] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:17.281447 31492 raft_consensus.cc:1261] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:17.283520 938 consensus_queue.cc:1035] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:17.303323 985 ts_tablet_manager.cc:927] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:17.309827 985 tablet_copy_client.cc:316] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.195:44831
I20260805 22:24:17.311722 31729 tablet_copy_service.cc:140] P feb5e7ab4b1541beaf2674c3c5500385: Received BeginTabletCopySession request for tablet e6cf90ae3fc840ceb48be8c790be97d9 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:58348)
I20260805 22:24:17.312264 31729 tablet_copy_service.cc:161] P feb5e7ab4b1541beaf2674c3c5500385: Beginning new tablet copy session on tablet e6cf90ae3fc840ceb48be8c790be97d9 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:58348: session id = 6bf8c694a0ba40978195ae044a917caf-e6cf90ae3fc840ceb48be8c790be97d9
I20260805 22:24:17.323050 31729 tablet_copy_source_session.cc:205] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:17.332054 985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6cf90ae3fc840ceb48be8c790be97d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:17.347687 985 tablet_copy_client.cc:791] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:17.348326 985 tablet_copy_client.cc:655] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:17.352802 985 tablet_copy_client.cc:531] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:17.361203 985 tablet_bootstrap.cc:492] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:17.381764 991 ts_tablet_manager.cc:927] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:24:17.382153 985 tablet_bootstrap.cc:492] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:17.382987 985 tablet_bootstrap.cc:492] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:17.383626 985 ts_tablet_manager.cc:1397] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.023s user 0.018s sys 0.004s
I20260805 22:24:17.386298 985 raft_consensus.cc:348] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:17.387104 985 raft_consensus.cc:729] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:17.387625 985 consensus_queue.cc:260] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:17.388238 31709 raft_consensus.cc:1261] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:17.389597 632 consensus_queue.cc:1035] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:17.390288 991 tablet_copy_client.cc:316] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:45607
I20260805 22:24:17.392469 31516 tablet_copy_service.cc:140] P f8e9f4c756da4203917c464e2b179994: Received BeginTabletCopySession request for tablet 91a9acdbc98a4e799374fe6e8950c3a7 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:52098)
I20260805 22:24:17.392963 31516 tablet_copy_service.cc:161] P f8e9f4c756da4203917c464e2b179994: Beginning new tablet copy session on tablet 91a9acdbc98a4e799374fe6e8950c3a7 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:52098: session id = 6bf8c694a0ba40978195ae044a917caf-91a9acdbc98a4e799374fe6e8950c3a7
I20260805 22:24:17.407931 985 ts_tablet_manager.cc:1428] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.024s user 0.004s sys 0.004s
I20260805 22:24:17.414914 31729 tablet_copy_service.cc:342] P feb5e7ab4b1541beaf2674c3c5500385: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-e6cf90ae3fc840ceb48be8c790be97d9 received from {username='slave'} at 127.0.0.1:58348
I20260805 22:24:17.415340 31729 tablet_copy_service.cc:433] P feb5e7ab4b1541beaf2674c3c5500385: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-e6cf90ae3fc840ceb48be8c790be97d9 on tablet e6cf90ae3fc840ceb48be8c790be97d9 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:17.418708 994 ts_tablet_manager.cc:927] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:24:17.419028 31516 tablet_copy_source_session.cc:205] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:17.421131 31492 raft_consensus.cc:1261] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:17.423724 632 consensus_queue.cc:1035] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:17.427160 991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a9acdbc98a4e799374fe6e8950c3a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:17.432430 994 tablet_copy_client.cc:316] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:45607
I20260805 22:24:17.438555 31516 tablet_copy_service.cc:140] P f8e9f4c756da4203917c464e2b179994: Received BeginTabletCopySession request for tablet c75e772a0f924080865ddf85ed1e2d86 from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:52098)
I20260805 22:24:17.439215 31516 tablet_copy_service.cc:161] P f8e9f4c756da4203917c464e2b179994: Beginning new tablet copy session on tablet c75e772a0f924080865ddf85ed1e2d86 from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:52098: session id = 6bf8c694a0ba40978195ae044a917caf-c75e772a0f924080865ddf85ed1e2d86
I20260805 22:24:17.452793 31516 tablet_copy_source_session.cc:205] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:17.457345 994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c75e772a0f924080865ddf85ed1e2d86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:17.458220 991 tablet_copy_client.cc:791] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:17.458773 991 tablet_copy_client.cc:655] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:17.467965 991 tablet_copy_client.cc:531] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:17.478336 994 tablet_copy_client.cc:791] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:17.479498 994 tablet_copy_client.cc:655] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:17.478395 991 tablet_bootstrap.cc:492] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:17.486848 994 tablet_copy_client.cc:531] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:17.495563 985 ts_tablet_manager.cc:927] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:24:17.496121 994 tablet_bootstrap.cc:492] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:17.508704 985 tablet_copy_client.cc:316] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.193:45607
I20260805 22:24:17.512917 31516 tablet_copy_service.cc:140] P f8e9f4c756da4203917c464e2b179994: Received BeginTabletCopySession request for tablet ab66683a9e5e46c2b09be4afdee2a08c from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:52098)
I20260805 22:24:17.513464 31516 tablet_copy_service.cc:161] P f8e9f4c756da4203917c464e2b179994: Beginning new tablet copy session on tablet ab66683a9e5e46c2b09be4afdee2a08c from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:52098: session id = 6bf8c694a0ba40978195ae044a917caf-ab66683a9e5e46c2b09be4afdee2a08c
I20260805 22:24:17.519411 991 tablet_bootstrap.cc:492] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:17.520216 991 tablet_bootstrap.cc:492] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:17.520840 991 ts_tablet_manager.cc:1397] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.043s user 0.027s sys 0.004s
I20260805 22:24:17.521275 31516 tablet_copy_source_session.cc:205] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:24:17.523892 991 raft_consensus.cc:348] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:17.524753 991 raft_consensus.cc:729] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:17.525434 985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab66683a9e5e46c2b09be4afdee2a08c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:24:17.525461 991 consensus_queue.cc:260] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:17.528033 991 ts_tablet_manager.cc:1428] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20260805 22:24:17.534029 31709 raft_consensus.cc:1261] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:17.535607 811 consensus_queue.cc:1035] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:17.557418 31492 raft_consensus.cc:1261] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:17.566390 31704 raft_consensus.cc:1261] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:17.568399 31516 tablet_copy_service.cc:342] P f8e9f4c756da4203917c464e2b179994: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-91a9acdbc98a4e799374fe6e8950c3a7 received from {username='slave'} at 127.0.0.1:52098
I20260805 22:24:17.568866 31516 tablet_copy_service.cc:433] P f8e9f4c756da4203917c464e2b179994: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-91a9acdbc98a4e799374fe6e8950c3a7 on tablet 91a9acdbc98a4e799374fe6e8950c3a7 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:17.569314 632 consensus_queue.cc:1035] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:17.582520 890 consensus_queue.cc:1035] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:17.584827 31492 raft_consensus.cc:1261] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:17.591789 902 consensus_queue.cc:1035] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:17.610308 985 tablet_copy_client.cc:791] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 0 data blocks...
I20260805 22:24:17.619333 994 tablet_bootstrap.cc:492] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:17.622136 985 tablet_copy_client.cc:655] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Starting download of 1 WAL segments...
I20260805 22:24:17.623843 994 tablet_bootstrap.cc:492] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:17.624611 994 ts_tablet_manager.cc:1397] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.129s user 0.044s sys 0.048s
I20260805 22:24:17.633671 994 raft_consensus.cc:348] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:17.634608 994 raft_consensus.cc:729] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:17.635272 994 consensus_queue.cc:260] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:17.639878 994 ts_tablet_manager.cc:1428] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.015s user 0.006s sys 0.002s
I20260805 22:24:17.642625 31516 tablet_copy_service.cc:342] P f8e9f4c756da4203917c464e2b179994: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-c75e772a0f924080865ddf85ed1e2d86 received from {username='slave'} at 127.0.0.1:52098
I20260805 22:24:17.643050 31516 tablet_copy_service.cc:433] P f8e9f4c756da4203917c464e2b179994: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-c75e772a0f924080865ddf85ed1e2d86 on tablet c75e772a0f924080865ddf85ed1e2d86 with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:17.642531 985 tablet_copy_client.cc:531] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260805 22:24:17.664189 985 tablet_bootstrap.cc:492] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf: Bootstrap starting.
I20260805 22:24:17.685109 985 tablet_bootstrap.cc:492] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260805 22:24:17.685967 985 tablet_bootstrap.cc:492] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf: Bootstrap complete.
I20260805 22:24:17.686611 985 ts_tablet_manager.cc:1397] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.008s
I20260805 22:24:17.689227 985 raft_consensus.cc:348] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:17.690088 985 raft_consensus.cc:729] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Initialized, Role: LEARNER
I20260805 22:24:17.690634 985 consensus_queue.cc:260] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:17.693032 985 ts_tablet_manager.cc:1428] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260805 22:24:17.696072 31516 tablet_copy_service.cc:342] P f8e9f4c756da4203917c464e2b179994: Request end of tablet copy session 6bf8c694a0ba40978195ae044a917caf-ab66683a9e5e46c2b09be4afdee2a08c received from {username='slave'} at 127.0.0.1:52098
I20260805 22:24:17.696475 31516 tablet_copy_service.cc:433] P f8e9f4c756da4203917c464e2b179994: ending tablet copy session 6bf8c694a0ba40978195ae044a917caf-ab66683a9e5e46c2b09be4afdee2a08c on tablet ab66683a9e5e46c2b09be4afdee2a08c with peer 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:17.727505 32587 raft_consensus.cc:1203] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:17.751760 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:17.795114 32587 raft_consensus.cc:1203] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:17.799400 31495 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:17.800110 31495 raft_consensus.cc:595] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:17.804054 31602 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
mode: GRACEFUL
new_leader_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:43438
I20260805 22:24:17.804620 31602 raft_consensus.cc:595] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Received request to transfer leadership to TS 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:17.807229 31709 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4172338662524cb29310f48e0861873e"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
mode: GRACEFUL
new_leader_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:43958
I20260805 22:24:17.807758 31709 raft_consensus.cc:595] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Received request to transfer leadership to TS 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:17.809717 32587 raft_consensus.cc:1203] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20260805 22:24:17.811343 31495 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:17.813694 31495 raft_consensus.cc:595] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:17.817615 31492 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:17.818149 31492 raft_consensus.cc:595] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:17.821470 31604 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
mode: GRACEFUL
new_leader_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:43438
I20260805 22:24:17.822019 31604 raft_consensus.cc:595] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Received request to transfer leadership to TS 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:17.824559 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 6
I20260805 22:24:17.826591 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:24:17.827195 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:17.902351 32587 raft_consensus.cc:1203] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20260805 22:24:17.923890 777 raft_consensus.cc:979] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c: : Instructing follower 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:17.924353 890 raft_consensus.cc:1067] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Signalling peer 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:17.925843 32587 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:57372
I20260805 22:24:17.926326 32587 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:17.926576 32587 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:17.931993 32587 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:17.933833 32587 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:17.934798 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:17.935395 31604 raft_consensus.cc:3032] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:17.935680 31604 raft_consensus.cc:729] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:17.935676 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:17.936321 31706 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:17.936509 31604 consensus_queue.cc:260] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:17.937793 31604 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:17.943573 31706 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:17.944674 32529 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:17.946732 790 raft_consensus.cc:2781] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:17.946854 31604 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:17.947171 790 raft_consensus.cc:686] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:17.949028 790 consensus_queue.cc:237] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:17.972749 31351 catalog_manager.cc:5495] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 2, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 2 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260805 22:24:18.342808 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 67437.7us
I20260805 22:24:18.508936 1021 raft_consensus.cc:979] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385: : Instructing follower 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:18.542810 878 raft_consensus.cc:1067] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Signalling peer 6bf8c694a0ba40978195ae044a917caf to start an election
W20260805 22:24:18.849020 938 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.583s user 0.000s sys 0.008s
W20260805 22:24:18.849609 938 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.584s user 0.000s sys 0.009s
I20260805 22:24:18.886876 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:19.096096 1035 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.096979 1035 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.099205 1035 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.113487 1026 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.114109 1026 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.116055 1026 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:19.137019 31669 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.721s user 0.004s sys 0.013s
W20260805 22:24:19.137465 31669 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.722s user 0.005s sys 0.013s
I20260805 22:24:19.137724 1035 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:19.139122 1024 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.138303 1035 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.139621 1024 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.142345 1024 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:19.148752 1035 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:19.158972 1026 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.159473 1028 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:19.159554 1026 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.160065 1028 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.161682 1026 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:19.161687 902 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:19.163376 902 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.165462 902 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.167176 1028 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:19.167906 1031 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.168434 1031 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
W20260805 22:24:19.171648 31668 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.813s user 0.008s sys 0.004s
W20260805 22:24:19.171972 31668 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.813s user 0.008s sys 0.004s
I20260805 22:24:19.173285 1028 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:19.173813 1028 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.176008 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.176252 1028 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.178159 1024 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.178669 1024 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:19.180570 1024 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:19.182060 1031 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.182883 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.185195 31706 raft_consensus.cc:1261] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:19.186599 31596 raft_consensus.cc:1261] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:19.187619 32587 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4172338662524cb29310f48e0861873e"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:57366
I20260805 22:24:19.188131 32587 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:19.188452 32587 raft_consensus.cc:3037] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.190078 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.192277 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.194146 1015 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:19.194675 1015 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.196605 1015 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:24:19.199316 31670 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.893s user 0.002s sys 0.010s
W20260805 22:24:19.199409 31667 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.875s user 0.005s sys 0.013s
W20260805 22:24:19.199734 31667 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.875s user 0.005s sys 0.013s
I20260805 22:24:19.200377 1047 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:19.200868 1047 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.202732 1047 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:19.204377 31670 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.898s user 0.002s sys 0.010s
I20260805 22:24:19.206673 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.207950 1022 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.208453 1022 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.208909 879 raft_consensus.cc:1050] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:19.211273 879 consensus_queue.cc:237] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.211297 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.213397 1022 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:19.197057 1015 consensus_queue.cc:1035] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.018s
I20260805 22:24:19.230484 1030 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:19.231209 1030 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.234596 1019 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.235250 1019 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.239245 1017 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:19.239809 1017 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.244388 1037 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.245044 1037 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.247278 1037 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.241650 1019 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:19.247705 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.249475 1017 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:19.283447 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.288393 32587 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.244325 1022 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.264233 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.298799 1022 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.299551 32591 raft_consensus.cc:1226] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:19.265197 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.265635 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.299742 32587 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.300774 1022 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:19.300911 1035 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:19.301468 1035 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.302944 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.271303 1030 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.239362 1040 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:19.293802 1028 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:19.294982 32594 raft_consensus.cc:1261] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:19.297353 1042 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:19.306665 1028 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.306880 1042 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:19.308390 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 114891us
I20260805 22:24:19.308754 1028 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.308933 1042 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:19.309275 1040 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.271517 1031 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:19.310632 1031 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.311340 1040 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:19.312594 1031 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.303717 1035 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.313326 1035 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.313867 1022 consensus_queue.cc:1046] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:19.313874 1035 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:19.315145 1031 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.315241 1022 raft_consensus.cc:3032] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:19.315580 1028 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:19.315757 1022 raft_consensus.cc:729] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:19.316098 1028 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.316118 879 consensus_queue.cc:1035] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:19.317829 1031 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.316855 1022 consensus_queue.cc:260] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.319047 1028 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.319365 1022 raft_consensus.cc:3037] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.319715 1031 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.320323 1028 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:19.320926 1028 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.322111 1015 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.322635 1015 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.323252 1028 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:19.324203 1035 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.324430 1015 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.325335 1015 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:19.325870 1015 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.337060 1047 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:19.327881 1015 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:19.337703 1027 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:19.327770 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.339017 1027 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.332640 1053 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:19.340502 1053 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.341765 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.342986 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.344775 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.346335 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.336568 1037 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:19.346997 1037 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.347510 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.348739 1015 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.349056 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.349447 1022 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.349310 1015 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.349862 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.349926 1022 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.350541 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.350780 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.351361 1015 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.352097 902 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:19.352345 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.330602 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.352607 902 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.353492 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:19.354067 31594 raft_consensus.cc:3037] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.354559 902 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.354769 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.351824 1047 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.356441 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.356541 1047 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.356632 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.342684 1053 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.359027 1022 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:19.359160 902 consensus_queue.cc:1035] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:19.359757 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.361940 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.363370 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.329679 1043 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:19.364178 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.364310 1043 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.365316 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.365856 31609 raft_consensus.cc:1261] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:19.366393 1043 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.366792 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.366438 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.368613 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.341140 1027 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.370046 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.371943 1055 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:19.352439 1037 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:19.374287 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.375274 32529 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.376109 902 raft_consensus.cc:2726] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.378206 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.378788 31711 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 1.
I20260805 22:24:19.379369 31594 raft_consensus.cc:2454] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:19.380000 31602 raft_consensus.cc:1226] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:19.380831 31711 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4172338662524cb29310f48e0861873e"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:43958
I20260805 22:24:19.380765 879 consensus_queue.cc:1035] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:19.381491 31711 raft_consensus.cc:595] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:19.381834 31711 raft_consensus.cc:603] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:24:19.372895 1055 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.383628 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.384400 31492 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:19.385023 31492 raft_consensus.cc:595] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 2 LEADER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.384487 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.386065 1055 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.386806 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.388339 31492 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "94d0d26905694943b7135a9b805cf106"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:19.388850 31492 raft_consensus.cc:595] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Received request to transfer leadership to TS 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:19.388907 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.331557 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:19.391546 31492 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:19.392073 31492 raft_consensus.cc:595] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:19.393472 902 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.393993 902 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.394797 31604 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
mode: GRACEFUL
new_leader_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:43438
I20260805 22:24:19.395396 31604 raft_consensus.cc:595] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Received request to transfer leadership to TS 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:19.395758 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.396878 902 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.400794 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:19.403435 31492 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:19.403650 1051 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.404007 31492 raft_consensus.cc:595] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:19.404114 1051 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.406459 1051 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:19.407033 1027 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:19.410452 1027 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.412891 1027 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.409188 1028 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:19.409643 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:19.413450 1058 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:19.413749 32587 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:24:19.413975 1058 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.414841 31668 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:19.415592 1051 raft_consensus.cc:2781] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:19.415917 1051 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.415982 1058 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.416313 1051 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.417397 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:19.418447 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:24:19.419250 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:19.419106 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:19.421089 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.421270 1028 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.423416 1028 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.429257 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:19.439767 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:19.440346 1037 raft_consensus.cc:979] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c: : Instructing follower 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:19.444320 1037 raft_consensus.cc:1067] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Signalling peer 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:19.441867 1035 raft_consensus.cc:979] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c: : Instructing follower 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:19.442572 31556 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.446672 1031 raft_consensus.cc:2726] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.447041 31556 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.448185 1031 raft_consensus.cc:2726] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.449009 1035 raft_consensus.cc:1067] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Signalling peer 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:19.453544 31704 raft_consensus.cc:2454] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:19.449856 32529 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:19.454620 902 raft_consensus.cc:2781] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:19.454981 902 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.455318 902 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.455943 1051 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.455298 879 raft_consensus.cc:2932] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:19.456458 31602 raft_consensus.cc:2932] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:19.460651 32529 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:19.461684 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:19.467402 902 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.469486 902 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.470516 32529 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.471112 902 raft_consensus.cc:2781] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:19.472047 902 raft_consensus.cc:686] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:19.472509 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.473176 1015 raft_consensus.cc:2726] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.474444 1051 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:19.474967 902 consensus_queue.cc:237] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.477581 31556 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.479552 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.480026 1035 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.479208 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:19.480526 1055 raft_consensus.cc:2726] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.480515 1035 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.481674 31711 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.482123 879 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.482502 1035 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.482661 879 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.484694 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.484941 879 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:19.485292 31494 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:19.485661 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:19.486881 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.487293 32594 raft_consensus.cc:2932] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:19.488364 31357 catalog_manager.cc:5495] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:19.492628 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:19.496690 32589 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:57372
I20260805 22:24:19.497228 32589 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:19.497535 32589 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.499305 31711 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:19.501317 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.503010 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.506373 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.504031 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.507237 31492 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:24:19.504637 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.505468 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.505731 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.508546 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:19.506582 31494 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:19.510538 32529 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:19.512341 31499 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:19.512465 32586 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:57372
I20260805 22:24:19.515901 32589 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.519682 1048 raft_consensus.cc:2781] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:19.520336 1048 raft_consensus.cc:686] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:19.521273 1048 consensus_queue.cc:237] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.513029 32586 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:19.523769 32589 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.525774 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.525206 32586 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.527562 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.529513 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.531205 1042 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:19.532327 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.531710 1042 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.536360 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.538034 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:19.538973 1042 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.540793 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.541875 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.542016 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.533552 1051 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.542687 1051 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.544593 1051 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:19.545696 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.556303 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:19.566835 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 165305us
I20260805 22:24:19.654132 1031 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:19.655288 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.655556 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.656026 31493 raft_consensus.cc:1261] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:19.656172 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.667820 1031 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:19.669137 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 156943us
I20260805 22:24:19.670061 1031 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:19.656533 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:19.584470 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.657215 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.655264 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.564431 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.656453 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.675235 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:19.675856 31602 raft_consensus.cc:3032] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:19.676132 31602 raft_consensus.cc:729] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:19.677031 31602 consensus_queue.cc:260] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.678989 31602 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.677515 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:19.681645 31704 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.691125 32586 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.693722 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.694952 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.698415 32589 raft_consensus.cc:1226] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:19.698541 31707 raft_consensus.cc:1226] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:19.699148 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.699901 32589 raft_consensus.cc:1226] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:19.699963 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.700842 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.701591 31359 catalog_manager.cc:5495] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 2, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 2 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:19.703310 31709 raft_consensus.cc:1226] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:19.703786 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.702389 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.705308 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.705868 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:19.707712 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:19.707723 32586 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:19.707899 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.680583 31554 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.709662 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.712606 1031 raft_consensus.cc:2726] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.712061 1035 consensus_queue.cc:1046] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:19.713788 1055 raft_consensus.cc:3032] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:19.714124 1055 raft_consensus.cc:729] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:19.714888 1055 consensus_queue.cc:260] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.716053 1055 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.716566 31554 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.719362 1031 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.719734 31602 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:19.721330 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:19.723143 31554 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.724357 32529 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.724476 31554 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.725051 1031 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:19.729797 31444 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.729861 31554 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.730772 1028 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.731093 1030 raft_consensus.cc:2781] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:19.731423 31444 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.731472 1030 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:19.730160 1031 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:19.725719 1037 raft_consensus.cc:2726] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.727494 1043 raft_consensus.cc:979] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994: : Instructing follower 516cb07ce3064ccbbca35e912d62816c to start an election
I20260805 22:24:19.732869 31554 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.733024 31444 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.733317 1043 raft_consensus.cc:1067] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Signalling peer 516cb07ce3064ccbbca35e912d62816c to start an election
I20260805 22:24:19.712723 31704 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:19.733963 31554 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.734117 31711 raft_consensus.cc:1226] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:19.734445 1037 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:19.734699 1035 raft_consensus.cc:2726] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.734817 1037 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:19.734159 31444 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.732218 1042 raft_consensus.cc:2726] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:19.729758 1073 raft_consensus.cc:2726] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.735211 1037 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:19.732116 1030 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.736095 1042 raft_consensus.cc:2726] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.737116 31444 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.725524 32529 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.734165 1043 raft_consensus.cc:2726] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.738761 1028 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.744442 31667 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:19.735132 31554 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.745217 31602 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:59028
I20260805 22:24:19.745292 32529 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.745790 31602 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:19.746145 31602 raft_consensus.cc:3037] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.745733 31554 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.747077 32529 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:19.748231 32529 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:19.749202 902 raft_consensus.cc:2726] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.750641 31444 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.750898 1043 consensus_queue.cc:1046] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20260805 22:24:19.752792 1076 raft_consensus.cc:2726] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.742803 1031 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:19.750659 32529 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:19.756222 1052 raft_consensus.cc:2726] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.754127 31554 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.749773 1048 raft_consensus.cc:2726] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.758738 1040 raft_consensus.cc:2726] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.759356 1047 raft_consensus.cc:3032] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:19.759706 1047 raft_consensus.cc:729] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:19.750963 1026 raft_consensus.cc:2726] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.760470 1047 consensus_queue.cc:260] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.762033 1048 raft_consensus.cc:2726] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:19.751266 1035 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:19.762563 1047 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.762637 1035 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:19.763135 1035 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
W20260805 22:24:19.766886 1078 log.cc:924] Time spent T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260805 22:24:19.750931 1022 raft_consensus.cc:2726] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.770561 32529 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
W20260805 22:24:19.771299 29789 debug-util.cc:398] Leaking SignalData structure 0x7b080000e660 after lost signal to thread 31668
I20260805 22:24:19.772473 1048 raft_consensus.cc:2781] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:19.772399 1042 raft_consensus.cc:2726] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:19.773051 1048 raft_consensus.cc:686] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:19.757223 1030 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.775843 1037 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.776705 1030 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.780741 31447 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.782054 31447 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.782423 31667 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:19.782548 31602 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:19.783650 1027 raft_consensus.cc:2726] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:19.788803 31602 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.788101 1048 consensus_queue.cc:237] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.785638 879 raft_consensus.cc:2726] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.785903 1061 raft_consensus.cc:2726] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.797667 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:19.800781 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:19.801151 1058 consensus_queue.cc:1046] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:19.802471 1061 raft_consensus.cc:979] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994: : Instructing follower 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:19.804051 1061 raft_consensus.cc:1067] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Signalling peer 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:19.812949 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:19.813567 31704 raft_consensus.cc:3037] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.812844 1042 raft_consensus.cc:1050] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
W20260805 22:24:19.825944 1042 raft_consensus.cc:1054] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994: Unable to promote non-voter 6bf8c694a0ba40978195ae044a917caf: Service unavailable: leader transfer in progress
I20260805 22:24:19.828446 1062 raft_consensus.cc:3032] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:19.828928 1062 raft_consensus.cc:729] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:19.809891 1037 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.845913 1062 consensus_queue.cc:260] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.845240 1035 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.854961 1035 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.857633 31601 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:19.860602 31704 raft_consensus.cc:2454] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
W20260805 22:24:19.861270 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 52085.6us
I20260805 22:24:19.861698 1086 raft_consensus.cc:1050] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:19.865651 32529 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:19.865289 1086 consensus_queue.cc:237] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.866575 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:19.866961 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:19.867805 31554 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:19.868589 1052 raft_consensus.cc:2781] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:19.868749 1048 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:19.869077 1052 raft_consensus.cc:686] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:19.862265 1031 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:19.870064 1052 consensus_queue.cc:237] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:19.871268 1031 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.874186 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:19.869786 1048 raft_consensus.cc:686] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:19.877547 1048 consensus_queue.cc:237] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:19.885953 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 92270.2us
W20260805 22:24:19.886232 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 102123us
I20260805 22:24:19.886767 31444 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.887584 31447 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:19.888579 1062 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.891848 31447 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:19.892129 1043 raft_consensus.cc:2726] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.893229 1043 raft_consensus.cc:1050] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
W20260805 22:24:19.893640 1043 raft_consensus.cc:1054] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994: Unable to promote non-voter 6bf8c694a0ba40978195ae044a917caf: Service unavailable: leader transfer in progress
I20260805 22:24:19.897547 1086 raft_consensus.cc:2726] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:19.897214 31444 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.910677 1058 raft_consensus.cc:2726] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:19.910879 1086 raft_consensus.cc:2726] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.912228 31444 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.912377 31667 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:19.913398 1086 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:19.914028 31667 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:19.916524 32586 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "94d0d26905694943b7135a9b805cf106"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:57382
I20260805 22:24:19.917124 32586 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:19.914601 31444 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.915386 1024 raft_consensus.cc:2726] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.917520 32586 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.917953 32589 raft_consensus.cc:1261] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:19.916501 31359 catalog_manager.cc:5495] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 1 to 2, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 2 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:19.928239 31444 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.926378 32586 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:19.930747 1047 raft_consensus.cc:2726] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.930794 1062 raft_consensus.cc:2726] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:19.932704 1062 raft_consensus.cc:979] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994: : Instructing follower feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:19.933204 1047 raft_consensus.cc:1067] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 2 LEADER]: Signalling peer feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:19.935357 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:19.936175 31707 raft_consensus.cc:2379] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 2: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:19.937424 32586 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:19.937654 1024 raft_consensus.cc:2726] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.941567 31357 catalog_manager.cc:5495] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 2, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 2 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:19.943979 31357 catalog_manager.cc:5495] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 2, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 2 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:19.947140 31707 raft_consensus.cc:1261] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:19.946800 31357 catalog_manager.cc:5495] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 2, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 2 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:19.949751 31709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:56260
I20260805 22:24:19.950258 31709 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:19.950578 31709 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:19.951910 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:19.953569 1024 consensus_queue.cc:1035] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:19.955297 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:19.953981 31667 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:19.957909 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:19.958247 1024 raft_consensus.cc:2726] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.958261 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:19.959582 31667 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:19.959859 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:19.960292 1024 raft_consensus.cc:2726] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.960413 31494 raft_consensus.cc:3032] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:19.960875 31494 raft_consensus.cc:729] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:19.961191 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:19.961364 31667 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:19.961598 31594 raft_consensus.cc:1261] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20260805 22:24:19.962091 1024 raft_consensus.cc:2726] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.963037 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:19.964104 31490 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.964459 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:19.964866 31707 raft_consensus.cc:1261] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:19.961612 31494 consensus_queue.cc:260] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:19.966383 1048 consensus_queue.cc:1035] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:19.968545 31554 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:19.966696 31494 raft_consensus.cc:3037] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:19.969759 1094 raft_consensus.cc:2726] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.970992 31709 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:19.963073 31667 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:19.975888 1024 raft_consensus.cc:2726] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.977025 31667 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:19.979187 31499 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:19.979499 31554 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.979650 31494 raft_consensus.cc:2454] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:19.980528 1094 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:19.981208 1062 consensus_queue.cc:1035] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:19.984445 1024 raft_consensus.cc:2726] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:19.987941 31667 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:19.989454 31490 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:19.994074 31601 raft_consensus.cc:1261] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:19.995666 1073 consensus_queue.cc:1035] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:19.997521 31554 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:19.999971 1047 consensus_queue.cc:1035] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:20.002642 1052 raft_consensus.cc:2726] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:20.001286 32529 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:20.000003 1024 raft_consensus.cc:2726] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:20.004622 1073 raft_consensus.cc:2781] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:20.005119 1073 raft_consensus.cc:686] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:20.005295 31709 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:20.005692 31554 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:20.006628 1052 raft_consensus.cc:2726] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:20.006894 1073 consensus_queue.cc:237] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:20.009888 31554 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:20.011332 1052 raft_consensus.cc:2726] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:20.025272 1062 raft_consensus.cc:2932] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:20.032249 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:20.032928 31499 raft_consensus.cc:3032] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:20.033221 31499 raft_consensus.cc:729] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:20.033977 31499 consensus_queue.cc:260] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:20.038203 31499 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:20.042142 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:20.042768 31596 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:20.050316 1043 consensus_queue.cc:1035] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:20.080441 31596 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:20.084609 32589 raft_consensus.cc:2932] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:20.095220 31602 raft_consensus.cc:1226] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:20.096766 31667 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:20.097918 31604 raft_consensus.cc:1226] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:20.100175 31499 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:20.112445 31596 raft_consensus.cc:1226] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:20.112828 879 raft_consensus.cc:2781] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:20.118237 879 raft_consensus.cc:686] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:20.137902 31357 catalog_manager.cc:5495] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 2, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 2 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:20.143023 1024 consensus_queue.cc:1046] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:20.145501 1024 raft_consensus.cc:3032] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:20.146190 1024 raft_consensus.cc:729] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:20.143272 31704 raft_consensus.cc:2932] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:20.164578 31596 raft_consensus.cc:2932] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:20.148068 1051 consensus_queue.cc:1046] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 2.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260805 22:24:20.148880 1022 consensus_queue.cc:1046] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.042s
I20260805 22:24:20.210830 1051 raft_consensus.cc:3032] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:20.226629 1051 raft_consensus.cc:729] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:20.227386 31707 raft_consensus.cc:1261] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:20.228036 1051 consensus_queue.cc:260] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:20.198314 31357 catalog_manager.cc:5495] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:20.185863 1024 consensus_queue.cc:260] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:20.230587 1024 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:20.220147 1022 raft_consensus.cc:3032] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:20.136076 879 consensus_queue.cc:237] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
W20260805 22:24:20.224486 31743 debug-util.cc:398] Leaking SignalData structure 0x7b08000b0740 after lost signal to thread 29786
I20260805 22:24:20.233592 1022 raft_consensus.cc:729] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
W20260805 22:24:20.236744 31743 debug-util.cc:398] Leaking SignalData structure 0x7b080014a7e0 after lost signal to thread 31402
I20260805 22:24:20.231199 1026 consensus_queue.cc:1035] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.006s
I20260805 22:24:20.237300 1022 consensus_queue.cc:260] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:20.215783 31490 raft_consensus.cc:2932] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:20.239802 1022 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
W20260805 22:24:20.238722 31743 debug-util.cc:398] Leaking SignalData structure 0x7b08001f9340 after lost signal to thread 31535
W20260805 22:24:20.258098 31743 debug-util.cc:398] Leaking SignalData structure 0x7b0800179e00 after lost signal to thread 31643
W20260805 22:24:20.259706 31743 debug-util.cc:398] Leaking SignalData structure 0x7b080035b3c0 after lost signal to thread 31748
W20260805 22:24:20.267206 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 124662us
W20260805 22:24:20.272882 31743 debug-util.cc:398] Leaking SignalData structure 0x7b080035de40 after lost signal to thread 32633
I20260805 22:24:20.297282 1051 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:20.353632 31494 raft_consensus.cc:1261] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260805 22:24:20.357098 1031 consensus_queue.cc:1035] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:20.409530 31704 raft_consensus.cc:1261] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260805 22:24:20.417774 31594 raft_consensus.cc:1261] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:20.448331 1093 consensus_queue.cc:1035] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:20.503539 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260805 22:24:20.511549 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 77381.1us
I20260805 22:24:20.473138 31486 raft_consensus.cc:1261] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:20.544687 31711 raft_consensus.cc:1261] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:20.558274 31707 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260805 22:24:20.569584 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 277534us
W20260805 22:24:20.571190 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 79758.8us
W20260805 22:24:20.576383 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 51302.9us
I20260805 22:24:20.644384 31359 catalog_manager.cc:5495] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 2 to 3, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 3 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20260805 22:24:20.686420 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 81156.5us
I20260805 22:24:20.691821 32587 raft_consensus.cc:3037] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 1 LEARNER]: Advancing to term 2
I20260805 22:24:20.697005 31494 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:20.698316 31494 raft_consensus.cc:595] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:20.670439 31601 raft_consensus.cc:1226] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:20.707957 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260805 22:24:20.702586 1136 consensus_queue.cc:1035] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260805 22:24:20.711549 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:24:20.712126 31596 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260805 22:24:20.715962 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 141878us
W20260805 22:24:20.722100 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 96944.9us
I20260805 22:24:20.722741 32587 raft_consensus.cc:1261] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 LEARNER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260805 22:24:20.737062 1143 consensus_queue.cc:1035] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:20.738021 1026 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:20.756096 31494 raft_consensus.cc:1261] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:20.757346 31490 raft_consensus.cc:1261] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:20.759639 1132 consensus_queue.cc:1035] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:20.768379 1026 consensus_queue.cc:1035] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:20.772511 879 consensus_queue.cc:1046] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:20.776547 1123 raft_consensus.cc:3032] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:20.777266 1123 raft_consensus.cc:729] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:20.776321 1143 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:20.781780 1123 consensus_queue.cc:260] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:20.800880 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 43494.2us
I20260805 22:24:20.795382 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:20.853871 1073 consensus_queue.cc:1035] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:20.854206 1143 consensus_queue.cc:1035] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:20.876549 32586 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEARNER]: Advancing to term 3
I20260805 22:24:20.902771 31495 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:24:20.915062 1138 raft_consensus.cc:1050] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:20.903302 1123 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:20.977471 31596 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:24:20.963301 1120 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20260805 22:24:20.941855 1138 consensus_queue.cc:237] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 2, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:21.067618 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 71336.8us
I20260805 22:24:21.119446 1147 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
W20260805 22:24:21.134766 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 75506.9us
I20260805 22:24:21.076107 1022 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.044s
W20260805 22:24:21.154460 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 150449us
I20260805 22:24:21.137629 1147 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:21.225214 31495 consensus_queue.cc:237] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:21.254007 1167 log.cc:924] Time spent T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.157s user 0.001s sys 0.000s
I20260805 22:24:21.258908 1147 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:21.305991 31705 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56246
I20260805 22:24:21.307333 31705 raft_consensus.cc:595] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Received request to transfer leadership
W20260805 22:24:21.282121 1169 log.cc:924] Time spent T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.096s user 0.002s sys 0.000s
W20260805 22:24:21.288220 1178 log.cc:924] Time spent T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.067s user 0.000s sys 0.002s
I20260805 22:24:21.320933 32586 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEARNER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:24:21.352062 1024 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:21.395032 879 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
W20260805 22:24:21.410538 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 239007us
I20260805 22:24:21.387553 1024 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:21.451587 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 235730us
W20260805 22:24:21.458621 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 252751us
I20260805 22:24:21.474714 31500 raft_consensus.cc:1261] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260805 22:24:21.475513 31707 raft_consensus.cc:1261] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
W20260805 22:24:21.515190 1123 consensus_meta.cc:287] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.541s user 0.016s sys 0.005s
I20260805 22:24:21.535195 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:21.546104 31494 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:21.573617 32586 raft_consensus.cc:1261] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 LEARNER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20260805 22:24:21.552579 1024 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:24:21.859308 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 254645us
W20260805 22:24:21.861755 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 272061us
W20260805 22:24:21.873816 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 287793us
I20260805 22:24:21.589908 1031 consensus_queue.cc:1035] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
W20260805 22:24:21.863853 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 273087us
I20260805 22:24:21.917366 32594 raft_consensus.cc:1261] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:21.916350 1170 consensus_queue.cc:1035] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
W20260805 22:24:21.975411 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 98052.6us
I20260805 22:24:21.953588 1132 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:21.950160 1086 consensus_queue.cc:1035] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260805 22:24:21.984529 1132 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:21.988754 1132 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:21.994763 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 121948us
I20260805 22:24:21.996739 31667 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:21.979698 1093 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:24:22.019212 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 91828.9us
I20260805 22:24:22.020874 1022 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:22.021752 1022 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:22.022461 1022 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:24:22.049501 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 51175.8us
I20260805 22:24:22.018519 1187 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:22.022631 1140 consensus_queue.cc:1035] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.027s
I20260805 22:24:22.012023 1093 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:22.046195 1198 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:24:22.088117 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 48807.4us
I20260805 22:24:22.092070 1122 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:22.092505 1179 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:22.106122 1197 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:22.082530 1187 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:22.122526 1173 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:22.097273 1198 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:22.150496 1093 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:22.134809 1197 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:22.155221 1187 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:22.155825 1197 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:22.127566 1179 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:22.161943 1191 log.cc:924] Time spent T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.256s user 0.006s sys 0.000s
I20260805 22:24:22.166044 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:22.127749 1122 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:22.200848 1122 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:22.161028 1179 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:22.212814 1190 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:22.230218 1216 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:22.231462 1216 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:22.247112 1185 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:22.232499 1198 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:22.253698 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 136953us
I20260805 22:24:22.210597 1173 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:22.237955 1179 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:22.254492 1199 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:22.287443 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:22.290750 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:22.292989 31444 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.156s user 0.023s sys 0.010s
W20260805 22:24:22.293682 31444 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.157s user 0.024s sys 0.010s
W20260805 22:24:22.294214 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 1.16136e+06us
I20260805 22:24:22.292774 1209 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:22.300266 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:22.303508 31494 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
W20260805 22:24:22.300963 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 198128us
I20260805 22:24:22.307613 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:22.265007 1216 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:22.248338 1185 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:22.325825 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:22.276259 1197 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:22.345134 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:22.348414 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:22.366096 1200 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:22.370855 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:22.272696 1179 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:22.373009 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 53789.5us
I20260805 22:24:22.316010 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:22.338418 1185 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:22.354996 1197 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:22.301595 1199 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:22.419255 1199 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:22.367427 1200 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:22.423264 1200 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:22.426798 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:22.413934 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:22.432222 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:22.433195 31596 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:22.435078 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:22.435209 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:22.388722 1093 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:22.447201 31706 raft_consensus.cc:1261] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:22.296474 1209 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:22.242283 1190 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:22.439802 31707 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:22.387761 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:22.447036 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:22.454427 1225 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:22.464541 1234 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:22.497998 31609 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:22.498842 1223 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:22.428754 1185 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:22.514150 1190 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:22.515375 1190 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:22.517669 31494 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:22.447582 1173 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:22.432147 1022 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:22.470934 1093 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:22.509173 1051 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:22.524621 1185 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:22.565929 1051 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:22.532792 31486 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40714 (request call id 1394) took 1083 ms. Trace:
I20260805 22:24:22.539734 1197 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:22.552686 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:22.516305 1190 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:22.558946 1209 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:22.560658 1022 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:22.561439 1179 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:22.468593 1232 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:22.563020 1223 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:22.581305 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 238115us
W20260805 22:24:22.584106 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 203725us
I20260805 22:24:22.589715 1051 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:22.604584 31554 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:22.585057 1185 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:22.569713 31486 rpcz_store.cc:276] 0805 22:24:21.449766 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:21.451210 (+ 1444us) service_pool.cc:227] Handling call
0805 22:24:22.531073 (+1079863us) spinlock_profiling.cc:243] Waited 118 ms on lock 0x7b5400331b54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fedaf0 00007f2731070f2a 00007f2735fadc6e 00007f272f6a4807 00007f272f6a4792 00007f272f6a46c1 00007f272f6a463c 00007f272f6a38f2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:22.532628 (+ 1555us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":271117184}
I20260805 22:24:22.591380 1232 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:22.663316 31559 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.510s user 0.066s sys 0.102s
I20260805 22:24:22.663360 1244 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
W20260805 22:24:22.681856 31559 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.579s user 0.067s sys 0.102s
I20260805 22:24:22.683766 1203 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:22.543298 1234 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:22.695051 1234 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:22.696539 31359 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47946 (request call id 180) took 1301 ms. Trace:
I20260805 22:24:22.689234 1232 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:22.679093 1212 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:22.705371 31359 rpcz_store.cc:276] 0805 22:24:21.396980 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:21.397281 (+ 301us) service_pool.cc:227] Handling call
0805 22:24:22.696385 (+1299104us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":130032,"reported_tablets":2}
I20260805 22:24:22.691087 1244 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:22.684900 1203 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:22.586977 1225 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:22.722863 1051 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:22.795320 1095 raft_consensus.cc:979] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994: : Instructing follower feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:22.796608 1235 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:22.798015 1229 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:22.797508 1051 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:22.798671 1229 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:22.799360 1229 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:22.779282 1093 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:22.710935 1212 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:22.808774 1051 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:22.811031 1239 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:22.817122 1223 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:22.818281 1223 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:22.801633 1190 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:22.812069 1239 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:22.819450 1223 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:22.823073 1212 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:22.870771 1225 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:22.797642 1235 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:22.922722 1239 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:22.914718 1234 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:22.916596 1185 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:22.918146 1076 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:22.937983 31707 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
mode: GRACEFUL
new_leader_uuid: "f8e9f4c756da4203917c464e2b179994"
from {username='slave'} at 127.0.0.1:43958
I20260805 22:24:22.939266 31707 raft_consensus.cc:595] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Received request to transfer leadership to TS f8e9f4c756da4203917c464e2b179994
I20260805 22:24:22.943813 1235 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:22.945433 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 264364us
I20260805 22:24:22.949366 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:22.933346 1239 raft_consensus.cc:1067] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Signalling peer feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:22.953895 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:22.952047 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:22.958648 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 495711us
I20260805 22:24:22.960124 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:22.960260 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260805 22:24:22.961622 31707 raft_consensus.cc:2454] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 1.
I20260805 22:24:22.970620 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:24:22.957923 1203 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:22.974140 1242 log.cc:924] Time spent T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.143s user 0.000s sys 0.002s
I20260805 22:24:22.973337 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:22.949095 1244 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:22.946539 1185 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:22.948475 1234 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.025219 1256 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:22.996160 1239 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:23.023834 1022 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:23.035123 1222 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:23.002596 1145 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:23.069464 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:23.028803 1223 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:23.061568 1212 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:22.948799 1076 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.035205 1185 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:23.063025 1051 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:23.076704 1239 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:23.101986 1051 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:23.103545 1234 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:23.104400 1239 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:23.072291 1022 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:23.075956 1209 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:23.107651 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:23.114449 32594 raft_consensus.cc:2454] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:24:23.116083 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:23.082351 1223 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:23.123838 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:23.109242 1239 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:23.109419 1243 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:23.072763 1256 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.109575 1225 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:23.104564 1222 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.158861 31667 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:23.162590 1222 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:23.168874 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:23.127357 1234 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:23.099974 1212 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.174405 31667 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:23.120697 1209 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:23.166206 1051 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:23.137042 1239 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:23.137306 1223 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.192369 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:23.152878 1225 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.180557 1222 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:23.077594 1145 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:23.212330 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 209973us
I20260805 22:24:23.175005 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:23.165900 1243 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.218045 1222 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:23.225333 1255 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:23.207710 1051 raft_consensus.cc:2726] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:23.186465 1234 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.234680 1222 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:23.241214 1131 raft_consensus.cc:979] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385: : Instructing follower f8e9f4c756da4203917c464e2b179994 to start an election
I20260805 22:24:23.241986 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:23.242447 1131 raft_consensus.cc:1067] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Signalling peer f8e9f4c756da4203917c464e2b179994 to start an election
I20260805 22:24:23.263250 32587 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:24:23.232919 1022 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:23.276461 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:23.248310 31597 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
W20260805 22:24:23.363983 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 354903us
I20260805 22:24:23.391446 32529 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:23.392287 31667 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:23.393450 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 170037us
I20260805 22:24:23.394186 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:23.396152 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:23.398846 31594 raft_consensus.cc:2379] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 3: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:23.402993 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:23.407007 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:23.235042 1255 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.405910 32591 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:24:23.410091 31594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59028 (request call id 1795) took 1021 ms. Trace:
I20260805 22:24:23.411968 31594 rpcz_store.cc:276] 0805 22:24:22.378296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:22.378509 (+ 213us) service_pool.cc:227] Handling call
0805 22:24:23.394023 (+1015514us) spinlock_profiling.cc:243] Waited 1.07 ms on lock 0x7b5800004598. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735faa1b0 00007f2735fa9a94 00007f272f691c04 00007f272f691b92
0805 22:24:23.399244 (+ 5221us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2462464}
I20260805 22:24:23.413092 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:23.415189 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:23.415843 31704 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 1.
I20260805 22:24:23.417217 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:23.417444 31556 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:23.420853 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:23.422735 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:23.329355 1280 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:23.389868 1212 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:23.291576 1076 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:23.337990 1209 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:23.403854 1048 raft_consensus.cc:2781] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:23.426976 1174 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:23.426849 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:23.405172 1131 raft_consensus.cc:2726] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:23.354228 1170 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:23.397735 1243 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:23.333894 1247 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:23.430835 1270 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
W20260805 22:24:23.437111 1196 log.cc:924] Time spent T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 1.487s user 0.014s sys 0.001s
I20260805 22:24:23.442761 1197 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:23.458820 1197 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:23.333673 1256 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:23.403548 1223 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:23.452278 1170 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.447818 1048 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:23.422986 1225 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:23.429404 1252 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:23.455157 31603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 839) took 1920 ms. Trace:
I20260805 22:24:23.455067 1247 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.430136 1239 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:23.435693 1212 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:23.460326 31444 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:23.448001 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:23.461582 1186 raft_consensus.cc:2781] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:23.471710 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:23.447008 1145 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:23.483033 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:23.464753 1048 raft_consensus.cc:3037] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:23.490134 1047 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:23.472196 1234 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:23.504316 1170 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:23.467507 1252 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.486112 1186 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:23.510646 1186 raft_consensus.cc:3037] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:23.491676 1047 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:23.478446 1212 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.481582 1239 consensus_queue.cc:1035] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.030s
I20260805 22:24:23.425271 1280 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.524755 1280 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:23.505451 31705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 1846) took 1071 ms. Trace:
I20260805 22:24:23.514030 1136 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:23.519866 1047 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
W20260805 22:24:23.527170 1229 consensus_meta.cc:287] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.723s user 0.016s sys 0.002s
W20260805 22:24:23.544389 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 125389us
I20260805 22:24:23.467707 31603 rpcz_store.cc:276] 0805 22:24:21.534773 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:21.538534 (+ 3761us) service_pool.cc:227] Handling call
0805 22:24:23.425254 (+1886720us) spinlock_profiling.cc:243] Waited 9.54 ms on lock 0x7b5800004598. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:24:23.430438 (+ 5184us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:23.454727 (+ 24289us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:23.454764 (+ 37us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:23.454815 (+ 51us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:24:23.454860 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:23.454932 (+ 72us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:23.455067 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21939968}
I20260805 22:24:23.528112 1252 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:23.472270 1174 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:23.565796 32529 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:23.467931 1270 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:23.544790 1229 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:23.540215 1136 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.566102 1252 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:23.636888 1252 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.640234 1252 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:23.571563 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 154220us
W20260805 22:24:23.643316 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 77599.5us
I20260805 22:24:23.643893 1184 raft_consensus.cc:2781] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260805 22:24:23.635944 1289 log.cc:924] Time spent T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.121s user 0.000s sys 0.001s
I20260805 22:24:23.645047 1235 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:23.643221 31667 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:23.644335 1184 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:23.660094 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:23.661113 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:23.662109 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:23.662380 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:23.663277 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:23.667656 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:23.669457 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:23.561667 1197 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:23.607128 1174 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:23.605881 1255 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:23.605180 1279 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:23.841303 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:23.778918 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 116377us
I20260805 22:24:23.834074 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:24:23.770373 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 289772us
I20260805 22:24:23.864708 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:23.835599 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:23.737298 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:23.778330 31596 raft_consensus.cc:2379] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 4: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
W20260805 22:24:23.873090 31556 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.196:41801 timed out after 1.675s (SENT) [suppressed 3 similar messages]
I20260805 22:24:23.874382 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:23.875147 31713 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:56260
I20260805 22:24:23.875764 31713 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:23.876161 31713 raft_consensus.cc:3037] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:23.876261 1070 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:23.877010 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:23.876848 1070 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.877668 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:23.873126 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:23.668303 31493 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:24:23.870256 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:23.875646 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:23.832367 1229 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:23.905999 1070 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:23.838315 1053 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:23.778767 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:23.908849 31495 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:23.633232 1212 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:23.917659 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:23.918313 31493 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:23.657716 1122 raft_consensus.cc:2726] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:23.528597 31705 rpcz_store.cc:276] 0805 22:24:22.434116 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:23.464732 (+1030616us) service_pool.cc:227] Handling call
0805 22:24:23.465107 (+ 375us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:23.474058 (+ 8951us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:23.477149 (+ 3091us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:23.479849 (+ 2700us) raft_consensus.cc:1631] Marking committed up to 1
0805 22:24:23.481309 (+ 1460us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:23.491293 (+ 9984us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:23.498786 (+ 7493us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:24:23.837683 1121 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:23.589717 1222 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:23.737217 1026 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
W20260805 22:24:23.929697 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 56638.5us
I20260805 22:24:23.657745 1184 raft_consensus.cc:3037] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:23.846931 1263 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:23.818322 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:23.817735 1131 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:23.937304 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:23.782714 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:23.925457 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:23.862095 31601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56384 (request call id 115) took 1005 ms. Trace:
I20260805 22:24:23.635250 1047 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:23.782578 1225 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:23.944983 31706 raft_consensus.cc:2354] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:24:23.841118 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:23.919723 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:23.930769 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:23.947311 31495 raft_consensus.cc:2454] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 1.
I20260805 22:24:23.877624 31715 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:23.910075 1053 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:23.917413 31708 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:24:23.929338 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:23.945749 31704 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:24:23.936538 1026 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.936659 1131 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:23.949214 1297 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:23.935925 1278 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:23.658085 1235 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.948962 31713 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.987877 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:23.939523 1212 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:23.956672 31502 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:23.937711 1174 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:23.934381 1121 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:23.941659 31601 rpcz_store.cc:276] 0805 22:24:22.856898 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:22.868988 (+ 12090us) service_pool.cc:227] Handling call
0805 22:24:22.915082 (+ 46094us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:22.925518 (+ 10436us) raft_consensus.cc:1520] Early marking committed up to index 5
0805 22:24:23.326830 (+401312us) log.cc:842] Serialized 14 byte log entry
0805 22:24:23.778591 (+451761us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:23.825073 (+ 46482us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:24:23.834007 (+ 8934us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:23.836263 (+ 2256us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:23.848428 (+ 12165us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":288450,"prepare.run_cpu_time_us":64,"prepare.run_wall_time_us":73,"raft.queue_time_us":108392,"raft.run_cpu_time_us":201,"raft.run_wall_time_us":39629,"thread_start_us":667298,"threads_started":2,"wal-append.queue_time_us":614085}
W20260805 22:24:23.987957 1298 log.cc:924] Time spent T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260805 22:24:24.018637 31602 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 1.
I20260805 22:24:24.010273 31713 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:24.013746 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:24.019560 32589 raft_consensus.cc:3037] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:23.999310 31486 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 1.
I20260805 22:24:24.006124 1140 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:24.016667 1311 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:24.029994 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.009348 1136 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:23.950098 1225 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.921974 1279 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:23.944464 1263 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:24.017838 1270 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.041522 31711 raft_consensus.cc:2932] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:24.053333 1308 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:24.059248 32589 raft_consensus.cc:2454] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:24.034718 1131 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:24.061882 1279 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.063771 1263 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.026072 1268 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:23.671502 1145 raft_consensus.cc:2932] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:24.069103 1263 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:24.015945 1297 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:24.016319 1174 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:24.017359 1212 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:24.068403 1047 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.070380 1222 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:24.018750 1278 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:24.070784 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:24.094026 1304 log.cc:924] Time spent T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260805 22:24:24.094255 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.084921 1026 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.085747 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.037299 1140 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:24.054018 1308 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:24.090884 1047 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:24.065570 1311 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:24.077817 1263 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:24.101621 1273 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:24.123675 1252 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:24.124279 1252 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:24.085093 1268 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:24.159945 31668 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:24.108911 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.151444 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.125497 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.149951 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:24.112663 1053 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:24.120656 1306 log.cc:924] Time spent T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.121s user 0.001s sys 0.000s
I20260805 22:24:24.163687 32586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57366 (request call id 230) took 1234 ms. Trace:
I20260805 22:24:24.126294 31705 raft_consensus.cc:1226] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:24.175884 879 raft_consensus.cc:2781] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:24.176326 879 raft_consensus.cc:686] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:24.153306 1174 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:24.177134 879 consensus_queue.cc:237] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:24.170307 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.172979 32586 rpcz_store.cc:276] 0805 22:24:22.929432 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:22.930411 (+ 979us) service_pool.cc:227] Handling call
0805 22:24:24.152818 (+1222407us) spinlock_profiling.cc:243] Waited 2.67 ms on lock 0x7b5800005198. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108c6c 00007f272e108bd6 000000000034aea9 00007f2735f34a30 00007f2735ffa7b4 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:24:24.152974 (+ 156us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:24.153158 (+ 184us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:24.153182 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:24.153226 (+ 44us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:24:24.153256 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:24.153311 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:24.163586 (+ 10275us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":6136064}
I20260805 22:24:24.179738 31713 raft_consensus.cc:2454] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:24:24.166183 31501 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 1.
I20260805 22:24:24.182348 31711 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
W20260805 22:24:24.167904 1186 consensus_meta.cc:287] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.657s user 0.006s sys 0.006s
I20260805 22:24:24.150776 1297 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:24.183904 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.152400 1305 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:24.184414 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:24.126245 1252 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.194113 31444 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:24.115265 1047 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:24.195962 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.173513 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:24.169145 1121 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:24.202670 1047 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:24.203107 1316 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:24.206991 1316 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:24.207373 1316 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:24.204276 31444 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:24.205675 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.185489 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.183766 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.212778 1047 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:24.214087 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.217080 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.213186 1047 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:24.220933 1047 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:24.221553 31600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56384 (request call id 113) took 1903 ms. Trace:
W20260805 22:24:24.222301 1048 consensus_meta.cc:287] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.723s user 0.016s sys 0.004s
I20260805 22:24:24.195772 1225 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.166958 1273 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:24.222724 1048 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:24.224783 31444 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:24.228461 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:24.205157 1308 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:24.212244 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.193694 1186 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:24.197809 1305 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:24.229179 31444 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:24.297622 31714 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 1851) took 1856 ms. Trace:
I20260805 22:24:24.290289 31709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1424) took 1976 ms. Trace:
I20260805 22:24:24.297122 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:24.299836 31356 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34394 (request call id 292) took 2065 ms. Trace:
W20260805 22:24:24.446861 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 217730us
W20260805 22:24:24.289049 31494 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1355) took 1981 ms (client timeout 2167 ms). Trace:
I20260805 22:24:24.434692 31714 rpcz_store.cc:276] 0805 22:24:22.441579 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:22.441825 (+ 246us) service_pool.cc:227] Handling call
0805 22:24:22.442333 (+ 508us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:22.571119 (+128786us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:22.571183 (+ 64us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:22.571280 (+ 97us) raft_consensus.cc:1631] Marking committed up to 1
0805 22:24:22.571348 (+ 68us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:24.297347 (+1725999us) spinlock_profiling.cc:243] Waited 128 ms on lock 0x7b54002b34d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:24.297409 (+ 62us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:24.297538 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":295295232}
I20260805 22:24:24.435564 31709 rpcz_store.cc:276] 0805 22:24:22.313921 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:24.289550 (+1975629us) service_pool.cc:227] Handling call
0805 22:24:24.289858 (+ 308us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:24.290015 (+ 157us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:24.290037 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:24.290071 (+ 34us) raft_consensus.cc:1631] Marking committed up to 1
0805 22:24:24.290096 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:24.290154 (+ 58us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:24.290239 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:24:24.438411 31356 rpcz_store.cc:276] 0805 22:24:22.235017 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:22.235350 (+ 333us) service_pool.cc:227] Handling call
0805 22:24:24.299774 (+2064424us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":2}
I20260805 22:24:24.449994 31444 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:24.450587 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:24.449048 31667 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:24.280212 1051 raft_consensus.cc:1050] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:24.449334 31501 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 4
I20260805 22:24:24.454627 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.459546 31709 raft_consensus.cc:2379] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:24.274789 1031 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:24.455315 1222 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:24.455832 1095 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:24.285030 31486 raft_consensus.cc:2396] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 3 because replica has last-logged OpId of term: 2 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 3.
I20260805 22:24:24.279323 1121 raft_consensus.cc:1050] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:24.281776 1131 raft_consensus.cc:1050] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:24.461100 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:24.228397 1251 raft_consensus.cc:2781] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260805 22:24:24.451601 31494 rpcz_store.cc:269] 0805 22:24:22.306456 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:22.307011 (+ 555us) service_pool.cc:227] Handling call
0805 22:24:24.287558 (+1980547us) spinlock_profiling.cc:243] Waited 207 ms on lock 0x7b5400122354. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:24.287695 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":475411712}
I20260805 22:24:24.227279 1235 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:24.388646 1263 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.289577 1273 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.459219 1291 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:24.226426 31715 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:24:24.466656 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.467741 1229 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:24.470798 1031 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:24.468981 1145 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:40714 (request call id 1393) took 3340 ms. Trace:
I20260805 22:24:24.292328 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:24.519703 1229 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:24.520224 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.519938 1291 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:24.521441 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.521751 1229 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.275250 1174 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:24.291168 1212 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:24.475414 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.456621 1270 consensus_queue.cc:1046] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:24.247249 1311 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:24.518287 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.359457 1184 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:24.468199 1316 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:24.484390 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.328979 1247 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:24.491508 32594 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57372 (request call id 418) took 1543 ms (client timeout 1675 ms). Trace:
I20260805 22:24:24.499383 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.531751 31708 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:24.533542 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:24.533982 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.534217 32579 raft_consensus.cc:2454] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:24:24.502388 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.528777 1174 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:24.509599 31667 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:24.533177 32594 rpcz_store.cc:269] 0805 22:24:22.948659 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:22.948858 (+ 199us) service_pool.cc:227] Handling call
0805 22:24:24.491321 (+1542463us) spinlock_profiling.cc:243] Waited 164 ms on lock 0x7b540007eed4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:24.491447 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":376575104}
I20260805 22:24:24.536448 31495 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:24.248443 1140 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:24.515002 31554 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:24.479569 1251 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:24.465282 1222 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:24.478484 1095 raft_consensus.cc:686] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:24.518882 31597 raft_consensus.cc:2379] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 2: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:24.522475 1291 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.454530 1319 raft_consensus.cc:2781] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:24.571700 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.571991 1319 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:24.572440 31602 raft_consensus.cc:2379] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 3: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:24.572371 1319 raft_consensus.cc:3037] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:24.449330 1268 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.470921 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.221978 31600 rpcz_store.cc:276] 0805 22:24:22.318458 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:22.331858 (+ 13400us) service_pool.cc:227] Handling call
0805 22:24:24.220844 (+1888986us) spinlock_profiling.cc:243] Waited 13.6 ms on lock 0x7b5800004598. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:24:24.221043 (+ 199us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:24.221215 (+ 172us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:24.221240 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:24.221282 (+ 42us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:24:24.221313 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:24.221364 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:24.221496 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":31192960}
W20260805 22:24:24.489494 32587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57366 (request call id 233) took 1321 ms (client timeout 1620 ms). Trace:
I20260805 22:24:24.529028 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:24.533727 1270 raft_consensus.cc:3032] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:24.504473 32529 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:24.501688 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.577529 31596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59030 (request call id 857) took 1451 ms. Trace:
I20260805 22:24:24.536545 31667 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:24.577768 31494 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:24.578604 31596 rpcz_store.cc:276] 0805 22:24:23.123185 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:23.123391 (+ 206us) service_pool.cc:227] Handling call
0805 22:24:24.574370 (+1450979us) spinlock_profiling.cc:243] Waited 653 ms on lock 0x7b54002391d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:24.574493 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1501928192}
I20260805 22:24:24.553328 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.562593 31447 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:24.528380 31556 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:24.534618 31500 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:24:24.569705 1251 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:24.567008 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.567193 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.583743 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.584156 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.584374 31492 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 1.
I20260805 22:24:24.563587 1273 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:24.519728 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.520319 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.529945 1048 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.590781 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:24.591949 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.592566 31500 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:24:24.577270 1270 raft_consensus.cc:729] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:24.593652 31705 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:24.594272 1270 consensus_queue.cc:260] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:24.595587 1270 raft_consensus.cc:3037] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:24.576934 1247 raft_consensus.cc:2781] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260805 22:24:24.573665 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 56550.3us
I20260805 22:24:24.538885 1206 raft_consensus.cc:2726] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:24.574743 31709 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:24.576930 31502 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 4
I20260805 22:24:24.578622 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.538707 32594 raft_consensus.cc:1226] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:24.580260 31447 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:24.581647 31667 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:24.581741 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 77345us
I20260805 22:24:24.602872 879 raft_consensus.cc:2726] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:24.603129 1095 consensus_queue.cc:237] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:24.603714 1186 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:24.604513 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:24.605409 31600 raft_consensus.cc:2379] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 3: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:24.606093 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.606371 1239 raft_consensus.cc:2726] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:24.606935 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:24.576737 32587 rpcz_store.cc:269] 0805 22:24:23.168002 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:23.168326 (+ 324us) service_pool.cc:227] Handling call
0805 22:24:24.489269 (+1320943us) spinlock_profiling.cc:243] Waited 92.9 ms on lock 0x7b5400503a54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:24.489412 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":213594368}
I20260805 22:24:24.608034 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.607204 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.610368 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.626710 32529 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:24.606536 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.674067 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.674664 1305 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:24.601773 1247 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:24.675431 1247 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:24.571606 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.569509 1222 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:24.569350 1132 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:24.691220 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:24.691766 1132 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:24.585901 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.693701 1132 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.586205 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.570013 1140 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:24.570331 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:24.572463 1031 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.519984 1145 rpcz_store.cc:276] 0805 22:24:21.122616 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:21.122866 (+ 250us) service_pool.cc:227] Handling call
0805 22:24:21.223687 (+100821us) spinlock_profiling.cc:243] Waited 28 ms on lock 0x7b58003374b0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2730fe3f63 00007f2731047aba 00007f2731063a7a 00007f2731064ada 00007f2731063c8b 00007f273106f084 00007f273106e743 00007f2731066c1a 00007f2735faa6d9 00007f272f694184
0805 22:24:21.471429 (+247742us) spinlock_profiling.cc:243] Waited 9.64 ms on lock 0x7b2c0003d328. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272fa4be11 00007f272fa5aada 00007f272fa5a763 00007f272fa5a701 00007f27310a3601 00007f2730ff53ee 00007f2731063ab9 00007f2731064ada 00007f2731063c8b 00007f273106f084 00007f273106e743
0805 22:24:21.486388 (+ 14959us) log.cc:842] Serialized 501 byte log entry
0805 22:24:23.433080 (+1946692us) spinlock_profiling.cc:243] Waited 4.73 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:24:23.657544 (+224464us) spinlock_profiling.cc:243] Waited 49.6 ms on lock 0x7b54001a7aa0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2730ffec33 00007f2731002665 00007f27310025f9 00007f2731002589 00007f2731002551 00007f273100184d 00007f27373a4204 00007f27373a4039 00007f272e1406b6 00007f272e143b11
0805 22:24:24.432510 (+774966us) log.cc:842] Serialized 14 byte log entry
0805 22:24:24.462587 (+ 30077us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":321056,"raft.queue_time_us":341137,"raft.run_cpu_time_us":2584,"raft.run_wall_time_us":49763,"spinlock_wait_cycles":211425408,"thread_start_us":100623,"threads_started":1,"wal-append.queue_time_us":436196}
W20260805 22:24:24.588685 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 60390.1us
I20260805 22:24:24.581269 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.586961 1273 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:24.580648 1197 raft_consensus.cc:2726] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:24.590013 1184 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.532452 1278 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.598305 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:24.585762 1255 raft_consensus.cc:2726] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:24.581501 1268 raft_consensus.cc:2726] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:24.599757 31554 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:24.600548 1276 raft_consensus.cc:2726] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:24.600888 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.673301 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.675273 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.675943 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:24.564224 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 101979us
I20260805 22:24:24.734673 31494 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:24:24.694221 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.708990 1316 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.596968 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:24.695185 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 113622us
I20260805 22:24:24.710726 31602 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
W20260805 22:24:24.695425 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 68808.8us
I20260805 22:24:24.737965 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.738422 31667 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:24.740927 31667 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:24.713059 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 132870us
I20260805 22:24:24.716460 1273 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.717406 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.782450 32529 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:24.718643 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.783895 32529 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:24.722397 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:24.785847 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 45937.3us
I20260805 22:24:24.674089 31600 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:24:24.726418 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:24.728601 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 128912us
I20260805 22:24:24.711807 1256 raft_consensus.cc:2726] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:24.735850 31604 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:24.712003 31501 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:24.739462 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:24.796079 32587 raft_consensus.cc:1226] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:24.796521 31494 raft_consensus.cc:2379] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:24.734632 1200 raft_consensus.cc:2781] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:24.779114 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:24:24.720474 1325 log.cc:924] Time spent T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.148s user 0.001s sys 0.000s
I20260805 22:24:24.725270 31603 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:24.786520 31444 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:24.742408 1328 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:24.786284 1206 raft_consensus.cc:2726] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:24.795305 1070 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:24.779138 1197 raft_consensus.cc:2726] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:24.792714 31554 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:24.800480 1200 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:24.794201 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.797035 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.770681 31667 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:24.841239 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 70670.2us
I20260805 22:24:24.842278 31667 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:24.843637 31667 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:24.803495 31501 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
from {username='slave'} at 127.0.0.1:40740
I20260805 22:24:24.804183 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.789896 1213 raft_consensus.cc:2781] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:24.805773 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.787989 32529 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:24.813509 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet c75e772a0f924080865ddf85ed1e2d86 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:24.789928 1251 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:24.785787 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.725253 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:24:24.885763 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 97839.9us
I20260805 22:24:24.886014 31711 raft_consensus.cc:3037] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:24.886332 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.814009 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:24.843724 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.891647 31608 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:24:24.806277 1174 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.893421 31502 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:24.894245 31711 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:24.894762 32581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57382 (request call id 442) took 1063 ms. Trace:
I20260805 22:24:24.895213 32581 rpcz_store.cc:276] 0805 22:24:23.831476 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:23.832692 (+ 1216us) service_pool.cc:227] Handling call
0805 22:24:23.841703 (+ 9011us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:24.077959 (+236256us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:24.080560 (+ 2601us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:24.084281 (+ 3721us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:24:24.085726 (+ 1445us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:24.883356 (+797630us) spinlock_profiling.cc:243] Waited 228 ms on lock 0x7b540017dc54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:24.884259 (+ 903us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:24.894707 (+ 10448us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":524225920}
W20260805 22:24:24.896930 1324 log.cc:924] Time spent T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.370s user 0.001s sys 0.011s
I20260805 22:24:24.723207 1140 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:24.898011 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.898715 1140 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:24.844520 31501 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
W20260805 22:24:24.881985 31339 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 82618us
I20260805 22:24:24.899262 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:24.900113 31711 raft_consensus.cc:2379] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:24.900025 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.815693 1246 raft_consensus.cc:2781] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:24.899322 31501 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:24.882088 31667 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:24.881232 31602 raft_consensus.cc:2454] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:24:24.832860 1070 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:24.901650 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.902433 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.917022 31713 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:24:24.844599 31554 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:24.908928 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.917429 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.909433 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.909871 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.916236 31501 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
W20260805 22:24:24.910969 1047 consensus_meta.cc:287] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.689s user 0.000s sys 0.011s
I20260805 22:24:24.911711 1199 raft_consensus.cc:2781] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:24.816133 1328 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:24.912588 32529 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:24.585583 1291 raft_consensus.cc:2932] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:24.913879 1246 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:24.880766 1213 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:24.900324 1206 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:24.909009 32579 raft_consensus.cc:2932] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
W20260805 22:24:24.918013 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 73483.7us
I20260805 22:24:24.841295 1273 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:24.910513 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.843844 1200 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:24.919919 879 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:24.929934 31486 raft_consensus.cc:2454] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:24.930570 31501 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:24.883777 1197 raft_consensus.cc:2726] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260805 22:24:24.931164 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 70301.8us
I20260805 22:24:24.932169 32529 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:24.916312 1258 raft_consensus.cc:2781] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:24.934652 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.946465 1048 raft_consensus.cc:2726] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:24.937870 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:24.918982 1070 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:24.919813 1140 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:24.931209 1199 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:24.941800 31554 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:24.944523 31594 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:24.945418 32529 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:24.958664 32529 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:24.959025 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 77028.2us
I20260805 22:24:24.959419 1184 raft_consensus.cc:2726] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:24.959723 32529 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:24.960166 31667 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:24.939342 1213 raft_consensus.cc:3037] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:24.961009 1256 raft_consensus.cc:2726] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:24.933652 1230 raft_consensus.cc:2781] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:24.942528 1273 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:24.940663 1206 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:24.948872 1258 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:24.938249 1246 raft_consensus.cc:3037] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:24.931326 1047 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:24.965371 31554 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:24.964521 1199 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:24.961081 31667 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:24.957233 1140 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:24.961212 1031 raft_consensus.cc:2781] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:24.919615 1319 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:24.964828 1048 raft_consensus.cc:2726] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:24.973860 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:24.957947 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.976048 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:24.967284 1230 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:24.987594 1230 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:24.974036 1174 raft_consensus.cc:2726] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:24.968927 1273 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:24.969434 1206 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:24.971700 1258 raft_consensus.cc:3037] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:24.963954 1187 consensus_queue.cc:1046] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260805 22:24:24.972503 32529 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:24.984714 31499 raft_consensus.cc:1226] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:24.975198 1216 raft_consensus.cc:2726] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:24.991170 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:24.974371 1095 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:24.978350 1140 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:24.994369 31495 raft_consensus.cc:2379] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 2: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:24.978155 1031 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:24.982935 1047 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:25.000286 31502 raft_consensus.cc:1226] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:25.000813 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.981402 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:25.009589 1255 raft_consensus.cc:2781] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:25.001405 1251 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:25.013692 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:25.015189 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:24.997263 1337 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:25.012418 1031 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:25.139153 31707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 1862) took 1264 ms. Trace:
I20260805 22:24:25.140473 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:25.017143 1326 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:25.011826 1328 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:25.145054 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:25.145550 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:25.146387 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 120342us
I20260805 22:24:25.145844 31707 rpcz_store.cc:276] 0805 22:24:23.874650 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:23.917431 (+ 42781us) service_pool.cc:227] Handling call
0805 22:24:23.917908 (+ 477us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:23.935454 (+ 17546us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:23.935500 (+ 46us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:23.935547 (+ 47us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:24:23.935581 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:25.032851 (+1097270us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b5400204754. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:25.138179 (+105328us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:25.139085 (+ 906us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":39919488}
I20260805 22:24:25.151053 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:25.154239 31494 raft_consensus.cc:2932] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:25.033900 1255 raft_consensus.cc:686] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:25.158439 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:25.009827 1095 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.167131 1339 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:25.167800 1339 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.170331 1339 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:25.172232 1258 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.173523 1336 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:25.173988 1336 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:25.174372 1336 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:25.154335 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:25.185392 31492 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
W20260805 22:24:25.229573 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 87245.3us
W20260805 22:24:25.234372 1332 log.cc:924] Time spent T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.257s user 0.001s sys 0.000s
I20260805 22:24:25.159493 1326 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:25.156388 1337 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.255513 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:25.234728 1319 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:25.199383 1070 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.271466 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:24:25.273155 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 107681us
W20260805 22:24:25.265175 1247 consensus_meta.cc:287] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.586s user 0.005s sys 0.006s
I20260805 22:24:25.277091 31554 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.277765 1342 raft_consensus.cc:2726] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.279071 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:25.280738 32529 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.281678 32591 raft_consensus.cc:1226] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:25.283193 1326 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:25.284417 31609 raft_consensus.cc:1226] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
W20260805 22:24:25.284335 31597 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56384 (request call id 121) took 1850 ms (client timeout 1518 ms). Trace:
I20260805 22:24:25.278062 31554 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.259845 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:25.286624 31554 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.288915 1256 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.287272 1336 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.290722 1336 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:25.288666 31554 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.292358 31554 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.293023 1344 raft_consensus.cc:2726] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.293865 1344 raft_consensus.cc:2726] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.294916 31554 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.296007 31554 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.295819 31444 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:25.272918 1258 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:25.300449 31447 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.299948 1344 raft_consensus.cc:2726] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:25.303464 1229 raft_consensus.cc:2726] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.307657 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:25.308338 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:25.310424 1290 raft_consensus.cc:2726] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.310750 31359 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47946 (request call id 181) took 2252 ms. Trace:
I20260805 22:24:25.310161 32583 raft_consensus.cc:2932] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:25.311110 31359 rpcz_store.cc:276] 0805 22:24:23.058860 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:23.059179 (+ 319us) service_pool.cc:227] Handling call
0805 22:24:25.310665 (+2251486us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27018,"reported_tablets":4}
I20260805 22:24:25.241236 1255 consensus_queue.cc:237] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.285121 1186 raft_consensus.cc:3032] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
W20260805 22:24:25.288035 1222 consensus_meta.cc:287] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.575s user 0.005s sys 0.004s
I20260805 22:24:25.300197 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:25.318703 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:25.319533 31493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52098 (request call id 111) took 1584 ms (client timeout 2004 ms). Trace:
I20260805 22:24:25.319669 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
W20260805 22:24:25.319873 31493 rpcz_store.cc:269] 0805 22:24:23.734811 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:23.926660 (+191849us) service_pool.cc:227] Handling call
0805 22:24:25.318572 (+1391912us) spinlock_profiling.cc:243] Waited 1.17 ms on lock 0x7b5800001898. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735faa1b0 00007f2735fa9a94 00007f272f691c04 00007f272f691b92
0805 22:24:25.319475 (+ 903us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2694400}
I20260805 22:24:25.320461 31499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1360) took 1660 ms. Trace:
I20260805 22:24:25.320828 31499 rpcz_store.cc:276] 0805 22:24:23.659653 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:23.666227 (+ 6574us) service_pool.cc:227] Handling call
0805 22:24:24.279794 (+613567us) spinlock_profiling.cc:243] Waited 5.94 ms on lock 0x7b5800001898. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108bce 000000000034aea9 00007f2735f34a30 00007f2735ffa7b4 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:24:24.279979 (+ 185us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:25.002773 (+722794us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:25.320217 (+317444us) spinlock_profiling.cc:243] Waited 700 ms on lock 0x7b54003a2854. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:25.320382 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1622804736}
I20260805 22:24:25.321774 31501 raft_consensus.cc:1226] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:25.322641 32529 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:25.299855 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 338825us
I20260805 22:24:25.301018 31444 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.324206 31667 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:25.325014 879 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:25.287793 1297 raft_consensus.cc:2726] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.322737 31554 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.326776 879 raft_consensus.cc:686] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:25.327630 879 consensus_queue.cc:237] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:25.285009 31597 rpcz_store.cc:269] 0805 22:24:23.435981 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:23.442165 (+ 6184us) service_pool.cc:227] Handling call
0805 22:24:25.284116 (+1841951us) spinlock_profiling.cc:243] Waited 986 ms on lock 0x7b540026fcd4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:25.284276 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2266902784}
I20260805 22:24:25.330415 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:25.332124 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:25.332564 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:25.332716 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:25.333249 31706 raft_consensus.cc:2379] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:25.334151 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:25.335033 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:25.335444 31707 raft_consensus.cc:1226] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:25.335391 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:25.335714 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:25.334197 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:25.337036 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:25.337786 31597 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:25.337093 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:25.338452 31711 raft_consensus.cc:2454] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:24:25.342633 31554 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:25.296118 1326 raft_consensus.cc:2726] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.345222 1337 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:25.345633 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:25.343778 31554 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.346837 1048 raft_consensus.cc:2726] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.347168 1326 raft_consensus.cc:2726] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.348817 31554 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.350092 31554 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.345947 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:25.344161 31444 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:25.351223 31554 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.318622 1190 raft_consensus.cc:2781] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:25.352495 31554 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.351776 1326 raft_consensus.cc:2726] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.353688 1276 raft_consensus.cc:2726] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.281561 1247 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.357161 31554 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.357359 31444 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.315045 1186 raft_consensus.cc:729] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:25.362001 31554 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.350708 1229 raft_consensus.cc:2726] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.349797 31600 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:25.363125 31444 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:25.350404 1095 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:25.352348 1251 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:25.367173 31444 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:25.360291 31667 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:25.359836 1192 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:25.352210 1211 raft_consensus.cc:2781] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:25.352002 1257 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.370509 1062 raft_consensus.cc:2781] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:25.376888 31444 leader_election.cc:398] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:24:25.365154 31554 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:25.361454 1213 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.358301 1297 raft_consensus.cc:2726] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.356798 1190 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:25.373634 31495 raft_consensus.cc:1226] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:25.375227 31667 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:25.356005 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:25.326831 1222 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.430445 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:25.430999 1222 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:25.431834 31708 raft_consensus.cc:2379] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:25.368150 879 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:25.368255 1339 raft_consensus.cc:979] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385: : Instructing follower f8e9f4c756da4203917c464e2b179994 to start an election
I20260805 22:24:25.364485 1312 raft_consensus.cc:2781] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:25.432591 1190 raft_consensus.cc:3037] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:25.348822 1031 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.319108 1239 consensus_queue.cc:1046] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.048s
I20260805 22:24:25.434036 1239 raft_consensus.cc:3032] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:25.434389 1239 raft_consensus.cc:729] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:25.435201 1239 consensus_queue.cc:260] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.364655 1326 raft_consensus.cc:2726] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.438891 1337 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:25.439541 1031 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:25.439407 1337 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.423662 1273 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.440762 31604 raft_consensus.cc:1226] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:25.425928 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:25.436430 1239 raft_consensus.cc:3037] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:25.389714 1174 raft_consensus.cc:2781] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:25.390093 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:25.390223 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:25.393671 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:25.369680 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:25.394572 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:25.387871 1095 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:25.441530 879 raft_consensus.cc:686] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:25.458334 1095 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:25.458410 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:25.458738 1095 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:25.459153 32586 raft_consensus.cc:1226] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:25.458817 879 consensus_queue.cc:237] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.459965 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:25.460852 32585 raft_consensus.cc:2379] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 2: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:25.384730 1062 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:25.399847 1247 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:25.369280 1340 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:25.401250 1140 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.377199 1211 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:25.404258 32594 raft_consensus.cc:1261] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:25.413363 1230 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.367973 1251 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:25.406100 31447 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.406523 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:25.401573 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:25.413829 31597 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:25.416512 1206 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.418066 1070 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:25.370944 1192 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:25.432647 1312 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:25.436702 31596 raft_consensus.cc:1226] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:25.387789 31444 leader_election.cc:401] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260805 22:24:25.462671 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
W20260805 22:24:25.445803 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 70643.7us
I20260805 22:24:25.453861 31494 raft_consensus.cc:1261] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:25.454681 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:25.376135 1199 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.370599 1276 consensus_queue.cc:1046] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.038s
W20260805 22:24:25.475378 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 69375.7us
I20260805 22:24:25.478842 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:25.446455 1022 raft_consensus.cc:2726] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.441628 1273 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:25.441332 1337 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:25.441908 1339 raft_consensus.cc:1067] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Signalling peer f8e9f4c756da4203917c464e2b179994 to start an election
I20260805 22:24:25.481077 31594 raft_consensus.cc:2354] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:24:25.468515 1062 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:25.446026 1174 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:25.468881 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:25.476740 1251 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
W20260805 22:24:25.476998 1200 consensus_meta.cc:287] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.531s user 0.009s sys 0.000s
I20260805 22:24:25.488747 32589 raft_consensus.cc:1226] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:25.474246 1211 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:25.481547 1213 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:25.499125 31609 raft_consensus.cc:1226] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:25.491245 31667 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:25.497289 1192 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:25.477252 1047 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.509936 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
W20260805 22:24:25.507175 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 131116us
I20260805 22:24:25.474138 1340 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.515600 31444 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:25.496594 1247 consensus_queue.cc:1046] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260805 22:24:25.516598 31444 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:25.516810 1206 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:25.513262 31667 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:25.517362 1247 raft_consensus.cc:3032] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:25.517517 1305 raft_consensus.cc:2781] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:25.517803 1247 raft_consensus.cc:729] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:25.496901 1312 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
W20260805 22:24:25.499552 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 94299.9us
W20260805 22:24:25.506016 1246 consensus_meta.cc:287] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.530s user 0.005s sys 0.009s
I20260805 22:24:25.510851 1270 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:24:25.521121 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:25.518096 31667 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:25.521883 31609 raft_consensus.cc:2379] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 2: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:25.505985 1174 raft_consensus.cc:3037] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:25.523211 1339 raft_consensus.cc:2726] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.522941 31667 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:25.523505 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:25.524052 31554 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.526407 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:25.522585 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:25.527334 879 raft_consensus.cc:2726] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.526904 1246 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.512789 31705 raft_consensus.cc:1226] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:25.532577 1230 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:25.530818 31554 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.531018 31444 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:25.515039 1022 raft_consensus.cc:2726] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.516614 1199 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:25.532419 31667 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:25.540515 1047 raft_consensus.cc:2726] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.489274 1349 consensus_queue.cc:1035] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.063s
I20260805 22:24:25.541534 1290 consensus_queue.cc:1046] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:25.543730 1273 raft_consensus.cc:2726] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.520865 1305 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:25.546991 1140 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:25.525306 1319 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:25.525595 1206 raft_consensus.cc:2726] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.544768 31554 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.545286 31667 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:25.547256 1305 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:25.546998 1290 raft_consensus.cc:3032] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:25.553185 31554 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:25.534106 1276 raft_consensus.cc:2726] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.518702 1247 consensus_queue.cc:260] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.554064 31667 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:25.529531 1031 consensus_queue.cc:1046] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260805 22:24:25.561532 1206 raft_consensus.cc:2726] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.519882 1200 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.562106 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:25.562901 31608 raft_consensus.cc:1226] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:25.562979 31490 raft_consensus.cc:2379] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 4: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:25.565853 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:25.568699 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:25.531705 1270 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:25.557981 31492 raft_consensus.cc:1226] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:25.557972 31554 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.547721 1339 raft_consensus.cc:2726] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.558655 1273 raft_consensus.cc:2726] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.560179 1247 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:25.549342 1340 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:25.571502 31554 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:25.583467 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:25.585235 31554 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.547458 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:25.576771 1276 raft_consensus.cc:2726] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.587178 1229 raft_consensus.cc:2726] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.588012 1186 consensus_queue.cc:260] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.557904 1022 raft_consensus.cc:3037] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:25.562155 1031 raft_consensus.cc:3032] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:25.589193 1229 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:25.577884 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:25.549850 1319 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:25.582626 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:25.557211 1290 raft_consensus.cc:729] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:25.571045 1200 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:25.556196 1140 raft_consensus.cc:2726] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.592213 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:25.595890 32589 raft_consensus.cc:1226] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:25.574324 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:25.600227 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:25.577102 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:25.638940 1186 raft_consensus.cc:3037] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:25.604774 1337 consensus_queue.cc:1046] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:25.647449 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:25.582582 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:25.650213 31714 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
W20260805 22:24:25.651013 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 108608us
I20260805 22:24:25.594063 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:25.653303 31495 raft_consensus.cc:2454] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:25.674592 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:25.656936 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:25.675709 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:25.676568 31495 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
from {username='slave'} at 127.0.0.1:40740
I20260805 22:24:25.678788 1246 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:25.679811 31604 raft_consensus.cc:1226] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:25.680389 1062 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.681761 31600 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:25.682260 1093 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:25.666275 1222 consensus_queue.cc:1046] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.017s
I20260805 22:24:25.303356 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:25.687312 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:25.683471 31493 raft_consensus.cc:1226] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:25.600862 1326 consensus_queue.cc:1046] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.057s
I20260805 22:24:25.677222 1022 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.691988 1326 raft_consensus.cc:3032] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:25.636869 1273 consensus_queue.cc:1046] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.058s
I20260805 22:24:25.697896 31357 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34400 (request call id 282) took 2188 ms. Trace:
I20260805 22:24:25.600402 1031 raft_consensus.cc:729] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:25.659709 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:25.570658 1211 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:25.662781 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 60097.8us
I20260805 22:24:25.578819 1270 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.664673 32529 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:25.710004 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 45403.7us
I20260805 22:24:25.710466 31714 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:25.576706 1095 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.651959 1319 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:25.651106 1229 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:25.669191 31444 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:25.636605 1342 consensus_queue.cc:1046] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.029s
I20260805 22:24:25.672379 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:25.727996 1342 raft_consensus.cc:3032] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:25.693521 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:25.695086 31495 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:25.728365 1342 raft_consensus.cc:729] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:25.728607 31495 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:25.696259 31609 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:25.729519 1342 consensus_queue.cc:260] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.730667 1342 raft_consensus.cc:3037] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:25.692374 1326 raft_consensus.cc:729] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:25.698282 31357 rpcz_store.cc:276] 0805 22:24:23.509122 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:23.509283 (+ 161us) service_pool.cc:227] Handling call
0805 22:24:25.697836 (+2188553us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":418605,"reported_tablets":3}
I20260805 22:24:25.702494 31704 raft_consensus.cc:1226] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
W20260805 22:24:25.655802 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 54053.7us
I20260805 22:24:25.707340 1273 raft_consensus.cc:3032] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:25.732390 1273 raft_consensus.cc:729] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:25.732137 1326 consensus_queue.cc:260] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.733328 1326 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:25.739284 32529 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:25.717042 1062 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:25.717480 1051 consensus_queue.cc:237] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 2, Current term: 3, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.703001 1022 raft_consensus.cc:3032] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:25.722194 1251 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.726895 1095 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:25.734079 31601 raft_consensus.cc:1226] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
W20260805 22:24:25.734582 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 65468.5us
I20260805 22:24:25.706584 31667 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:25.741184 31554 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.742807 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:25.744462 1022 raft_consensus.cc:729] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:25.744354 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:25.751678 31494 raft_consensus.cc:2379] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 2: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:25.751405 1022 consensus_queue.cc:260] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.745123 32529 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:25.752599 31494 raft_consensus.cc:1226] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:25.714649 1048 raft_consensus.cc:2726] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.742043 1047 raft_consensus.cc:2781] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:25.746641 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:25.746807 1184 consensus_queue.cc:1046] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:24:25.747133 31709 raft_consensus.cc:1226] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:25.751551 31715 raft_consensus.cc:2932] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 2 to 4, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:25.736620 1229 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:25.754516 31554 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.755275 31444 leader_election.cc:398] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
W20260805 22:24:25.755795 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 49309.9us
I20260805 22:24:25.756554 31495 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.753409 32529 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:25.765976 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:25.759295 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:25.745947 1255 raft_consensus.cc:3037] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:25.773459 31667 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:25.773650 31707 raft_consensus.cc:2379] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:25.697707 1290 consensus_queue.cc:260] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.782490 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:25.767964 1270 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:25.777494 31495 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:25.761152 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:25.778723 1258 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:25.779430 1211 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:25.748962 1031 consensus_queue.cc:260] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.791033 1251 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:25.776357 31667 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:25.792634 1339 raft_consensus.cc:2781] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:25.792973 1339 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:25.793357 1339 raft_consensus.cc:3037] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:25.768569 1276 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:25.785969 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:25.771571 1200 raft_consensus.cc:2726] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.779989 1226 consensus_queue.cc:1046] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:25.789393 1297 consensus_queue.cc:1046] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:25.789721 31705 raft_consensus.cc:2379] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:25.785609 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:25.801358 31596 raft_consensus.cc:2354] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 2. Current term is 3.
I20260805 22:24:25.762908 1184 raft_consensus.cc:3032] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:25.768270 1213 raft_consensus.cc:2726] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.759143 1337 raft_consensus.cc:2726] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.797719 31601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 1818) took 1210 ms. Trace:
I20260805 22:24:25.796666 1062 consensus_queue.cc:1046] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.004s
I20260805 22:24:25.798941 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:25.781904 1349 raft_consensus.cc:2726] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.759534 1203 consensus_queue.cc:1046] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.037s
I20260805 22:24:25.802765 1197 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:25.802944 1297 raft_consensus.cc:3032] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:25.803910 1200 raft_consensus.cc:3032] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:25.804211 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:25.835675 31714 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:25.807435 1340 consensus_queue.cc:1046] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:25.809201 31554 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:25.809904 31601 rpcz_store.cc:276] 0805 22:24:24.587727 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:25.447181 (+859454us) service_pool.cc:227] Handling call
0805 22:24:25.447488 (+ 307us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:25.749318 (+301830us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:25.797308 (+ 47990us) spinlock_profiling.cc:243] Waited 286 ms on lock 0x7b54001bba54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:25.797660 (+ 352us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":658654592}
I20260805 22:24:25.810207 31444 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.776492 1047 raft_consensus.cc:686] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:25.814996 31356 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34394 (request call id 293) took 1341 ms. Trace:
I20260805 22:24:25.844533 31356 rpcz_store.cc:276] 0805 22:24:24.474066 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:24.474199 (+ 133us) service_pool.cc:227] Handling call
0805 22:24:25.814931 (+1340732us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":269565,"reported_tablets":3}
I20260805 22:24:25.733256 1273 consensus_queue.cc:260] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.800995 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:25.818073 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:25.810868 1184 raft_consensus.cc:729] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:25.823284 31495 raft_consensus.cc:1226] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:25.823680 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:25.824411 31494 raft_consensus.cc:1226] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:25.797269 1258 raft_consensus.cc:686] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:25.830018 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:25.832978 31667 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:25.852988 31667 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:25.842519 31554 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.849243 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:25.856943 31667 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:25.858007 31667 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:25.862074 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:25.829491 1048 raft_consensus.cc:3032] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:25.818248 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:25.861766 1179 raft_consensus.cc:2726] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.862677 31554 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:25.882882 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 72738.4us
I20260805 22:24:25.883524 31714 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:25.884263 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:25.825649 1297 raft_consensus.cc:729] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:25.841226 1200 raft_consensus.cc:729] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:25.848342 1291 raft_consensus.cc:2726] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.819226 1222 consensus_queue.cc:1035] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260805 22:24:25.888890 31444 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.905668 31554 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.858946 1337 raft_consensus.cc:2726] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:25.862566 1186 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.870299 32584 raft_consensus.cc:2379] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 2: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:25.859175 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:25.841144 1197 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:25.905714 1356 log.cc:924] Time spent T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.254s user 0.001s sys 0.000s
I20260805 22:24:25.916339 31714 raft_consensus.cc:1226] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:25.886482 1263 raft_consensus.cc:2726] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.882853 1048 raft_consensus.cc:729] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:25.922267 31554 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:25.887705 1226 raft_consensus.cc:2726] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.887358 1206 raft_consensus.cc:2726] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.889032 1336 raft_consensus.cc:2726] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:25.862876 1276 consensus_queue.cc:1046] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.049s
I20260805 22:24:25.948279 31709 raft_consensus.cc:1226] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:25.843412 31708 consensus_queue.cc:237] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.912228 1291 raft_consensus.cc:2726] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:25.908941 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:25.897038 1257 consensus_queue.cc:1046] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.034s
I20260805 22:24:25.929826 1062 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.906976 1216 consensus_queue.cc:1046] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.044s
I20260805 22:24:25.902952 1190 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.968716 1197 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:25.968954 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:25.969825 31705 raft_consensus.cc:2354] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 2. Current term is 3.
I20260805 22:24:25.970320 31709 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "42324773ce144eb784e14693eecbee72"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:43958
I20260805 22:24:25.990667 31709 raft_consensus.cc:595] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:25.962062 31600 raft_consensus.cc:2379] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 2: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:25.974452 1257 raft_consensus.cc:3032] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:25.976687 1190 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:25.978036 1363 log.cc:924] Time spent T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20260805 22:24:25.956422 1047 consensus_queue.cc:237] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
W20260805 22:24:25.958549 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 52645.5us
I20260805 22:24:25.984977 31554 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:25.957710 1247 raft_consensus.cc:2726] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:25.926893 1184 consensus_queue.cc:260] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:25.972993 1192 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:26.023362 1366 raft_consensus.cc:3032] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:26.020635 31351 catalog_manager.cc:5495] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New cstate: current_term: 1 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:26.004395 31490 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 3
I20260805 22:24:26.027935 31713 raft_consensus.cc:3032] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Stepping down as leader of term 3
W20260805 22:24:26.031597 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 71808.3us
I20260805 22:24:26.033329 31444 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:26.047660 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260805 22:24:25.977268 31499 raft_consensus.cc:1226] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:26.002806 1257 raft_consensus.cc:729] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:26.054641 31502 raft_consensus.cc:1226] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:25.955456 1297 consensus_queue.cc:260] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:26.050097 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 166603us
I20260805 22:24:25.957712 1258 consensus_queue.cc:237] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:26.052831 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:24:26.059544 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:26.046207 1328 raft_consensus.cc:2781] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:26.060503 1328 raft_consensus.cc:686] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:26.061610 1328 consensus_queue.cc:237] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:25.993340 1048 consensus_queue.cc:260] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:26.065127 1349 raft_consensus.cc:3032] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:26.076454 1349 raft_consensus.cc:729] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:25.957115 1200 consensus_queue.cc:260] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:26.078132 1200 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:26.077950 1349 consensus_queue.cc:260] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:26.069250 1297 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:26.051107 31713 raft_consensus.cc:729] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:26.070938 31351 catalog_manager.cc:5495] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 2 to 3, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
W20260805 22:24:26.082489 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 49214.7us
I20260805 22:24:26.083436 31444 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:26.085156 31444 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:26.088333 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:26.093626 31609 raft_consensus.cc:1261] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:26.094264 31495 raft_consensus.cc:1261] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:26.089085 31604 raft_consensus.cc:2362] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4. Re-sending same reply.
I20260805 22:24:26.103590 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:26.104211 31601 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:26.104554 31609 raft_consensus.cc:1226] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:26.048277 1366 raft_consensus.cc:729] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:26.111419 31444 leader_election.cc:398] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 2. Current term is 3.
I20260805 22:24:26.111892 31444 leader_election.cc:401] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260805 22:24:26.116328 31602 raft_consensus.cc:1226] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:26.119963 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:26.027241 1305 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:26.090665 1062 raft_consensus.cc:2726] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:26.091929 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
W20260805 22:24:26.126893 1312 consensus_meta.cc:287] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.608s user 0.000s sys 0.011s
I20260805 22:24:26.082419 1326 consensus_queue.cc:1046] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.070s
I20260805 22:24:26.056905 1247 consensus_queue.cc:1046] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.069s
I20260805 22:24:26.168888 879 consensus_queue.cc:1046] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:26.096379 1369 raft_consensus.cc:2726] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:26.205574 31493 raft_consensus.cc:1226] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:26.109336 1095 raft_consensus.cc:2781] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:26.140170 1319 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:26.163604 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:26.160038 31501 raft_consensus.cc:2379] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 2: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
W20260805 22:24:26.164248 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 107976us
W20260805 22:24:26.229427 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 55060.6us
I20260805 22:24:26.165537 31444 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:26.235323 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 69873.1us
I20260805 22:24:26.268355 31494 raft_consensus.cc:1226] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:26.269536 1184 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:26.272617 31444 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:26.273998 31444 leader_election.cc:398] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 2. Current term is 3.
I20260805 22:24:26.127625 1251 consensus_queue.cc:1046] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.113s
I20260805 22:24:26.159373 1350 raft_consensus.cc:3032] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:26.286229 1381 raft_consensus.cc:2726] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
W20260805 22:24:26.274333 1174 consensus_meta.cc:287] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.742s user 0.006s sys 0.006s
I20260805 22:24:26.271716 1273 raft_consensus.cc:3037] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:26.265669 1239 raft_consensus.cc:2726] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:26.166432 1312 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:26.285497 1350 raft_consensus.cc:729] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:26.284757 1095 raft_consensus.cc:686] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 2 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:26.393359 31444 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:26.376245 31713 consensus_queue.cc:260] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.4, Last appended by leader: 4, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:26.300884 1022 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:26.306941 1336 raft_consensus.cc:3032] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:26.159917 1257 consensus_queue.cc:260] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:26.384573 32586 raft_consensus.cc:1226] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:26.383831 31600 raft_consensus.cc:1261] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:26.402824 1192 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:26.403782 1336 raft_consensus.cc:729] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:26.325140 1174 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:26.404140 1062 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:26.455996 1174 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:26.407835 1239 consensus_queue.cc:1046] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.032s
I20260805 22:24:26.451789 1312 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:26.475914 31603 raft_consensus.cc:1226] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:26.457235 1350 consensus_queue.cc:260] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:26.478137 1350 raft_consensus.cc:3037] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:26.435827 1290 raft_consensus.cc:3037] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
W20260805 22:24:26.480031 1229 consensus_meta.cc:287] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.691s user 0.004s sys 0.008s
W20260805 22:24:26.482358 1342 consensus_meta.cc:287] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.751s user 0.008s sys 0.005s
W20260805 22:24:26.497293 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 200328us
I20260805 22:24:26.501914 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:26.481695 1186 raft_consensus.cc:3032] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:26.491839 1305 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:26.559170 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 402768us
I20260805 22:24:26.506299 1319 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:26.508430 1186 raft_consensus.cc:729] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:26.506856 1229 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:26.564611 1031 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
W20260805 22:24:26.578191 1378 log.cc:924] Time spent T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.261s user 0.000s sys 0.002s
I20260805 22:24:26.581851 32583 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEARNER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 3 index: 4. (index mismatch)
W20260805 22:24:26.582885 1339 consensus_meta.cc:287] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.789s user 0.009s sys 0.003s
W20260805 22:24:26.583277 1255 consensus_meta.cc:287] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.802s user 0.006s sys 0.007s
I20260805 22:24:26.583854 1255 raft_consensus.cc:2726] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:26.583561 1339 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:26.502517 1095 consensus_queue.cc:237] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:26.586473 1366 consensus_queue.cc:260] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:26.590919 1190 raft_consensus.cc:3032] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:26.591373 1190 raft_consensus.cc:729] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:26.554859 1230 consensus_queue.cc:1035] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.019s
I20260805 22:24:26.591897 1186 consensus_queue.cc:260] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:26.592319 1190 consensus_queue.cc:260] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:26.593194 1186 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:26.598255 1339 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:26.599294 1349 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:26.602010 1140 raft_consensus.cc:3032] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:26.541116 1336 consensus_queue.cc:260] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:26.599255 1190 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:26.625626 1336 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:26.636027 1366 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:24:26.679642 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 277521us
I20260805 22:24:26.681423 31499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40740 (request call id 902) took 1307 ms. Trace:
I20260805 22:24:26.682130 31499 rpcz_store.cc:276] 0805 22:24:25.373335 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:25.374975 (+ 1640us) service_pool.cc:227] Handling call
0805 22:24:25.389675 (+ 14700us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:26.064316 (+674641us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:26.680451 (+616135us) spinlock_profiling.cc:243] Waited 587 ms on lock 0x7b54001fde54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:26.681092 (+ 641us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1350783744}
W20260805 22:24:26.683004 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 134087us
W20260805 22:24:26.689663 31492 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.521s user 0.009s sys 0.017s
I20260805 22:24:26.684998 31596 raft_consensus.cc:1226] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:26.603164 1140 raft_consensus.cc:729] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:26.692931 31713 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:26.704453 1140 consensus_queue.cc:260] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:26.707844 1140 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:26.565706 31667 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:26.711324 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 145549us
W20260805 22:24:26.681627 31447 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.529s user 0.000s sys 0.007s
W20260805 22:24:26.700335 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 150342us
W20260805 22:24:26.722395 31492 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.553s user 0.010s sys 0.018s
I20260805 22:24:26.728259 1048 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
W20260805 22:24:26.725510 31447 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.573s user 0.000s sys 0.013s
I20260805 22:24:26.735783 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:26.740931 32579 raft_consensus.cc:2379] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 2: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:26.742863 1337 raft_consensus.cc:465] T f914c8243e724eda849ce1fc2dc30827 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Not starting pre-election -- already a leader
I20260805 22:24:26.744351 1229 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:26.561640 31705 raft_consensus.cc:1261] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:26.750617 31609 raft_consensus.cc:1261] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:26.759181 1377 raft_consensus.cc:2726] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:26.768896 1337 consensus_queue.cc:1035] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:26.775831 1257 raft_consensus.cc:3037] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:26.791607 31601 raft_consensus.cc:1261] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260805 22:24:26.806977 31713 raft_consensus.cc:2396] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for term 4 because replica has last-logged OpId of term: 3 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260805 22:24:26.809623 1379 consensus_queue.cc:1035] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
W20260805 22:24:26.800395 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 79883.3us
I20260805 22:24:26.814664 1203 consensus_queue.cc:1035] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:26.817713 1273 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:26.826303 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:26.824950 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:26.838070 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:26.845157 31708 raft_consensus.cc:1226] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 1. Current term is 2. Ops: []
W20260805 22:24:26.847146 31351 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.607s user 0.009s sys 0.008s
I20260805 22:24:26.848376 31711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:56246 (request call id 800) took 1363 ms. Trace:
I20260805 22:24:26.849565 31711 rpcz_store.cc:276] 0805 22:24:25.485277 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:25.485486 (+ 209us) service_pool.cc:227] Handling call
0805 22:24:26.847520 (+1362034us) spinlock_profiling.cc:243] Waited 1.33 s on lock 0x7b5400205654. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fedaf0 00007f2731070f2a 00007f2735fadc6e 00007f272f6a4807 00007f272f6a4792 00007f272f6a46c1 00007f272f6a463c 00007f272f6a38f2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:26.848178 (+ 658us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3053572864}
W20260805 22:24:26.852347 1131 raft_consensus.cc:1054] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385: Unable to promote non-voter 6bf8c694a0ba40978195ae044a917caf: Illegal state: Replica feb5e7ab4b1541beaf2674c3c5500385 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
W20260805 22:24:26.852653 31351 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.733s user 0.010s sys 0.008s
I20260805 22:24:26.843521 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:24:26.856912 31715 consensus_meta.cc:287] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.979s user 0.010s sys 0.010s
I20260805 22:24:26.821256 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
W20260805 22:24:26.866570 31490 consensus_meta.cc:287] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.792s user 0.014s sys 0.006s
W20260805 22:24:26.865960 1121 raft_consensus.cc:1054] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385: Unable to promote non-voter 6bf8c694a0ba40978195ae044a917caf: Illegal state: Replica feb5e7ab4b1541beaf2674c3c5500385 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } } pending_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
W20260805 22:24:26.871691 1184 consensus_meta.cc:287] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.592s user 0.010s sys 0.008s
I20260805 22:24:26.867023 31490 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:26.873688 31490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1401) took 1131 ms. Trace:
I20260805 22:24:26.873723 1263 raft_consensus.cc:2726] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 2. Current term is 3.
I20260805 22:24:26.875478 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:26.874541 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:26.882300 1273 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:26.879935 31490 rpcz_store.cc:276] 0805 22:24:25.741946 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:25.744328 (+ 2382us) service_pool.cc:227] Handling call
0805 22:24:25.965771 (+221443us) spinlock_profiling.cc:243] Waited 6.32 ms on lock 0x7b5800001898. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735faa1b0 00007f2735fa9a94 00007f272f691c04 00007f272f691b92
0805 22:24:26.873576 (+907805us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":14537088}
I20260805 22:24:26.889240 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:26.881903 1200 consensus_meta.cc:287] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.803s user 0.007s sys 0.012s
I20260805 22:24:26.892527 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:26.896400 31713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56260 (request call id 1889) took 1317 ms. Trace:
I20260805 22:24:26.899286 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:26.899466 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:26.900317 31714 raft_consensus.cc:2379] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 3: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:26.896706 31713 rpcz_store.cc:276] 0805 22:24:25.573984 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:25.574091 (+ 107us) service_pool.cc:227] Handling call
0805 22:24:26.317202 (+743111us) spinlock_profiling.cc:243] Waited 12.4 ms on lock 0x7b5400278420. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2730ffe951 00007f273105f321 00007f2731060b4e 00007f273106c228 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2
0805 22:24:26.690457 (+373255us) spinlock_profiling.cc:243] Waited 61 ms on lock 0x7b580049e5b0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2730fe3f63 00007f2731047aba 00007f273105f3af 00007f2731060b4e 00007f273106c228 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c
0805 22:24:26.891294 (+200837us) spinlock_profiling.cc:243] Waited 374 ms on lock 0x7b5400205654. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:26.891392 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1029861888}
I20260805 22:24:26.909539 1200 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:26.910097 1200 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:26.912110 31711 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:26.912330 1200 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:26.914316 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:26.905543 31596 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:26.920469 31603 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:24:26.920760 31609 raft_consensus.cc:1226] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260805 22:24:26.921685 1273 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:26.915165 31711 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:26.927031 31554 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:26.927886 1200 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:26.928298 1200 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:26.928663 1200 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:26.936995 31707 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:26.901907 1263 consensus_queue.cc:1046] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:26.945520 31714 raft_consensus.cc:1226] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260805 22:24:26.945353 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:26.951141 1121 consensus_queue.cc:1035] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:26.944217 1263 raft_consensus.cc:3032] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:26.956341 31713 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 3 index: 4. Preceding OpId from leader: term: 4 index: 4. (term mismatch)
I20260805 22:24:26.956768 31713 pending_rounds.cc:76] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385: Aborting all ops after (but not including) 3
I20260805 22:24:26.956997 31713 pending_rounds.cc:97] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385: Aborting uncommitted CHANGE_CONFIG_OP operation due to leader change: 3.4
I20260805 22:24:26.957355 31713 raft_consensus.cc:2894] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Aborting config change with OpId 3.4: Aborted: Op aborted by new leader
I20260805 22:24:26.957530 31707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56260 (request call id 1902) took 1126 ms. Trace:
I20260805 22:24:26.957476 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:26.958540 31707 rpcz_store.cc:276] 0805 22:24:25.830950 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:25.849052 (+ 18102us) service_pool.cc:227] Handling call
0805 22:24:26.957159 (+1108107us) spinlock_profiling.cc:243] Waited 1.04 s on lock 0x7b54001e2154. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:26.957284 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2395728256}
I20260805 22:24:26.961601 31713 raft_consensus.cc:2866] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:24:26.962389 31554 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:26.967932 31444 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:26.968580 31667 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:26.970494 1062 raft_consensus.cc:2781] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:26.971118 1062 raft_consensus.cc:686] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:26.972218 1062 consensus_queue.cc:237] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:26.954780 1263 raft_consensus.cc:729] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:26.976128 31711 raft_consensus.cc:2379] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 3: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:26.983239 1290 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.008s
I20260805 22:24:26.985863 31490 raft_consensus.cc:1226] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:26.986953 1070 consensus_queue.cc:1046] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.022s
I20260805 22:24:26.988251 1070 raft_consensus.cc:3032] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:26.988457 31492 raft_consensus.cc:1226] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:26.988754 1022 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:26.989177 1022 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:26.989557 1022 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:27.001669 1263 consensus_queue.cc:260] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:27.003199 1263 raft_consensus.cc:3037] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:27.003978 31492 raft_consensus.cc:1226] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:26.978024 1257 raft_consensus.cc:2726] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:26.997570 1070 raft_consensus.cc:729] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:27.014870 1070 consensus_queue.cc:260] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.016309 1070 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:26.955669 1200 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:27.020155 1121 consensus_queue.cc:1046] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:27.035444 1199 consensus_queue.cc:1035] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:27.037954 1200 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:27.042115 1048 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:27.042869 1048 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.042274 31492 raft_consensus.cc:1261] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20260805 22:24:27.042081 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:27.045590 31711 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 4
I20260805 22:24:27.050392 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:27.050902 31492 raft_consensus.cc:1226] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:27.051271 31603 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 1.
I20260805 22:24:27.051999 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:27.052722 31490 raft_consensus.cc:1261] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260805 22:24:27.069154 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:27.057476 1379 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:27.073221 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:27.075021 31713 raft_consensus.cc:1261] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260805 22:24:27.062464 1048 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:27.060389 31715 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1470) took 1309 ms. Trace:
I20260805 22:24:27.077765 31707 raft_consensus.cc:2396] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf for term 3 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260805 22:24:27.092119 1062 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:27.085032 31600 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:27.109489 31711 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:27.080576 1379 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.114969 31554 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:27.115636 1291 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:27.092252 1140 consensus_queue.cc:1035] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.012s
I20260805 22:24:27.121277 1200 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election won for term 4
W20260805 22:24:27.130416 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 130252us
I20260805 22:24:27.112516 1062 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.136979 31444 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:27.142409 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:27.143411 1022 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:27.147477 31492 raft_consensus.cc:1261] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260805 22:24:27.140853 1200 raft_consensus.cc:686] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:27.151296 1379 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:27.151239 1095 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:27.152376 1095 raft_consensus.cc:686] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:27.153065 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:27.153584 1095 consensus_queue.cc:237] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:27.156008 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:27.156580 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:27.156422 1200 consensus_queue.cc:237] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:27.157310 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:27.135828 1291 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.159430 1291 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:27.156749 31494 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:27.080813 31715 rpcz_store.cc:276] 0805 22:24:25.748698 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:25.748837 (+ 139us) service_pool.cc:227] Handling call
0805 22:24:25.751253 (+ 2416us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:25.751448 (+ 195us) raft_consensus.cc:1520] Early marking committed up to index 4
0805 22:24:26.895360 (+1143912us) log.cc:842] Serialized 14 byte log entry
0805 22:24:26.980007 (+ 84647us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:27.018352 (+ 38345us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:24:27.028887 (+ 10535us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:27.039699 (+ 10812us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:27.057632 (+ 17933us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":23043,"raft.queue_time_us":21536,"raft.run_cpu_time_us":224,"raft.run_wall_time_us":15276,"thread_start_us":57615,"threads_started":1,"wal-append.queue_time_us":124793}
I20260805 22:24:27.153569 31447 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:27.164865 31444 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:27.168076 1062 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:27.154405 1131 consensus_queue.cc:1046] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260805 22:24:27.170080 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:27.170557 1131 raft_consensus.cc:3032] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:27.170967 1131 raft_consensus.cc:729] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:27.171867 1131 consensus_queue.cc:260] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.171545 31502 raft_consensus.cc:1226] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 2 LEADER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:27.173028 1349 raft_consensus.cc:3032] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:27.174674 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:27.174963 1257 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:27.175482 1257 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.177266 31556 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:27.178056 1140 raft_consensus.cc:2726] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:27.178574 1257 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:27.173472 1349 raft_consensus.cc:729] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:27.180481 1349 consensus_queue.cc:260] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:27.181440 1349 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:27.181571 31600 raft_consensus.cc:1226] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:27.184664 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:27.184834 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:27.185585 32579 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:27.188807 1022 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:27.190065 31556 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:27.190618 31499 raft_consensus.cc:2454] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:27.174675 1381 raft_consensus.cc:2726] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260805 22:24:27.191507 1411 log.cc:924] Time spent T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.147s user 0.005s sys 0.000s
I20260805 22:24:27.203127 31490 raft_consensus.cc:2454] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:27.193576 1291 raft_consensus.cc:2781] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:27.207480 1263 raft_consensus.cc:2726] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:27.173344 1131 raft_consensus.cc:3037] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
W20260805 22:24:27.207618 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 54165us
I20260805 22:24:27.208186 1291 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:27.213537 1291 raft_consensus.cc:3037] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:27.214395 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:27.215991 1190 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:27.216645 1190 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.218739 1190 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:27.219995 31444 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:27.223345 1062 raft_consensus.cc:2726] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:27.224840 31444 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:27.224993 31600 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:27.226706 1263 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:27.227283 1263 raft_consensus.cc:686] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:27.228380 1263 consensus_queue.cc:237] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:27.233808 31444 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:27.238586 31667 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:27.244591 1121 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:27.296777 1350 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:27.297832 1350 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:27.306512 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 72870.4us
I20260805 22:24:27.305357 1121 raft_consensus.cc:686] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:27.307864 1350 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:27.308216 1070 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:27.310599 1095 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:27.312119 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:27.312279 1190 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260805 22:24:27.315167 1070 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.316799 1121 consensus_queue.cc:237] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:27.314689 31602 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:27.319661 31667 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:27.321590 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:27.322263 31707 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:27.322742 1350 raft_consensus.cc:2781] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:27.327139 1350 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:27.327533 1350 raft_consensus.cc:3037] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:27.327922 1095 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.320415 1257 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:27.330263 1257 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:27.331446 32529 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:27.332250 1379 raft_consensus.cc:2781] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:27.332686 1379 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:27.317258 1070 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:27.334045 1379 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:27.332378 1257 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:27.343745 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:27.350840 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 60401.5us
I20260805 22:24:27.352905 31502 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:27.357182 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:27.357978 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:27.349789 1070 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:27.366297 31602 raft_consensus.cc:2379] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 3: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:27.354835 1190 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:27.357632 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:27.375070 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:27.378379 31501 raft_consensus.cc:1226] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
W20260805 22:24:27.380152 1439 log.cc:924] Time spent T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.088s user 0.002s sys 0.000s
I20260805 22:24:27.389871 31596 raft_consensus.cc:2379] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 3: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:27.380569 1190 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.375653 1095 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:27.394744 1397 raft_consensus.cc:2781] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:27.420404 1397 raft_consensus.cc:686] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 2 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:27.421681 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:27.421555 1397 consensus_queue.cc:237] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.392318 1070 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:27.426302 1443 log.cc:924] Time spent T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.125s user 0.001s sys 0.000s
I20260805 22:24:27.432585 1190 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:27.433394 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:27.434669 1376 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:27.435171 32584 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEARNER]: Advancing to term 4
I20260805 22:24:27.436823 1376 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.438263 31351 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34388 (request call id 277) took 2509 ms. Trace:
I20260805 22:24:27.440168 1376 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:27.440357 31581 tablet_service.cc:1514] Processing DeleteTablet for tablet c75e772a0f924080865ddf85ed1e2d86 with delete_type TABLET_DATA_TOMBSTONED (TS 516cb07ce3064ccbbca35e912d62816c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59024
I20260805 22:24:27.442191 1070 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:27.444734 1200 consensus_queue.cc:1035] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:27.434803 1379 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.446694 1379 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:27.447532 32584 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEARNER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:24:27.449051 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:27.449836 32579 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:27.454582 1131 consensus_queue.cc:1046] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260805 22:24:27.458621 1131 raft_consensus.cc:3032] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:27.455493 31554 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:27.458981 1131 raft_consensus.cc:729] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:27.466498 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:27.466423 1131 consensus_queue.cc:260] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:27.467813 1131 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:27.469758 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:27.470479 31711 raft_consensus.cc:2454] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:27.451267 1291 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:27.475850 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 50930.8us
I20260805 22:24:27.477443 1291 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:27.474100 31502 raft_consensus.cc:1261] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:27.482587 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:27.483283 32584 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:27.483354 32529 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:27.485592 31554 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:27.485952 1257 raft_consensus.cc:2726] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:27.486085 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:27.486689 1257 raft_consensus.cc:2781] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:27.486788 32584 raft_consensus.cc:3037] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:27.475090 1350 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:27.487212 1257 raft_consensus.cc:686] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:27.490621 1213 raft_consensus.cc:2781] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:27.490995 1213 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:27.491300 1213 raft_consensus.cc:3037] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:27.496361 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:27.497014 31597 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:27.497393 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:27.498515 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:27.498780 32529 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:27.438617 31351 rpcz_store.cc:276] 0805 22:24:24.928666 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:24.928833 (+ 167us) service_pool.cc:227] Handling call
0805 22:24:27.438159 (+2509326us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0805 22:24:26.885902 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0805 22:24:26.888764 (+ 2862us) write_op.cc:432] Acquiring schema lock in shared mode
0805 22:24:26.888818 (+ 54us) write_op.cc:435] Acquired schema lock
0805 22:24:26.888877 (+ 59us) tablet.cc:660] Decoding operations
0805 22:24:26.889597 (+ 720us) write_op.cc:620] Acquiring the partition lock for write op
0805 22:24:26.889720 (+ 123us) write_op.cc:641] Partition lock acquired for write op
0805 22:24:26.889779 (+ 59us) tablet.cc:683] Acquiring locks for 2 operations
0805 22:24:26.963427 (+ 73648us) tablet.cc:699] Row locks acquired
0805 22:24:26.963467 (+ 40us) write_op.cc:260] PREPARE: finished
0805 22:24:26.963720 (+ 253us) write_op.cc:270] Start()
0805 22:24:26.963902 (+ 182us) write_op.cc:276] Timestamp: P: 1785968666963641 usec, L: 0
0805 22:24:26.963940 (+ 38us) op_driver.cc:331] REPLICATION: starting
0805 22:24:26.964967 (+ 1027us) log.cc:842] Serialized 924 byte log entry
0805 22:24:27.211618 (+246651us) op_driver.cc:444] REPLICATION: finished
0805 22:24:27.243317 (+ 31699us) write_op.cc:301] APPLY: starting
0805 22:24:27.243415 (+ 98us) tablet.cc:1365] starting BulkCheckPresence
0805 22:24:27.243583 (+ 168us) tablet.cc:1368] finished BulkCheckPresence
0805 22:24:27.243775 (+ 192us) tablet.cc:1370] starting ApplyRowOperation cycle
0805 22:24:27.298581 (+ 54806us) tablet.cc:1381] finished ApplyRowOperation cycle
0805 22:24:27.299734 (+ 1153us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0805 22:24:27.310957 (+ 11223us) write_op.cc:312] APPLY: finished
0805 22:24:27.349967 (+ 39010us) log.cc:842] Serialized 29 byte log entry
0805 22:24:27.363345 (+ 13378us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:27.366877 (+ 3532us) write_op.cc:454] Released schema lock
0805 22:24:27.372990 (+ 6113us) write_op.cc:341] FINISH: Updating metrics
0805 22:24:27.409813 (+ 36823us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:27.411210 (+ 1397us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":749553,"reported_tablets":6,"child_traces":[["op",{"apply.queue_time_us":31402,"apply.run_cpu_time_us":3558,"apply.run_wall_time_us":181430,"num_ops":2,"prepare.queue_time_us":766131,"prepare.run_cpu_time_us":74640,"prepare.run_wall_time_us":192956,"raft.queue_time_us":51129,"raft.run_cpu_time_us":11412,"raft.run_wall_time_us":92329,"replication_time_us":247582,"thread_start_us":920438,"threads_started":4,"wal-append.queue_time_us":157300}]]}
I20260805 22:24:27.500913 1257 consensus_queue.cc:237] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:27.502105 1048 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:27.504992 31711 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:27.511484 32529 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:27.513281 31667 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:27.508608 1022 consensus_queue.cc:1046] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260805 22:24:27.516607 1372 consensus_queue.cc:1035] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:27.517297 1022 raft_consensus.cc:3032] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:27.517668 1022 raft_consensus.cc:729] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:27.518440 1022 consensus_queue.cc:260] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:27.519778 1022 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:27.511631 31356 catalog_manager.cc:5495] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 2, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 2 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:27.524725 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:27.525244 1140 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:27.525838 1140 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:27.525619 31356 catalog_manager.cc:5495] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 2, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:27.526453 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:27.527258 31501 raft_consensus.cc:2379] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 2: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:27.527084 31356 catalog_manager.cc:5495] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 2 to 4, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:27.525655 31490 raft_consensus.cc:2379] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 2: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:27.530081 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:27.530627 1376 raft_consensus.cc:2781] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:27.530716 31499 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:27.531100 1376 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:27.531502 1376 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:27.527763 1140 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:27.543665 31711 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:27.545419 1062 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:27.545644 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:27.546013 1062 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.546362 31490 raft_consensus.cc:2362] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3. Re-sending same reply.
I20260805 22:24:27.541991 32529 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:27.547982 32529 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:27.548384 1048 raft_consensus.cc:2726] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:27.548707 1070 raft_consensus.cc:2781] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:27.549042 1048 consensus_queue.cc:1035] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:27.549268 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:27.549957 31490 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:27.551761 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:27.551874 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:27.552554 31490 raft_consensus.cc:2454] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:27.553133 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:27.553692 31492 raft_consensus.cc:3037] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:27.554411 1366 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:24:27.548007 1062 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:27.556535 31447 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:27.511672 32584 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:27.549208 1070 raft_consensus.cc:686] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:27.557641 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:27.558233 31494 raft_consensus.cc:2454] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:27.558244 1070 consensus_queue.cc:237] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.558676 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:27.559424 31597 raft_consensus.cc:2454] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:27.560436 1121 raft_consensus.cc:2781] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:27.561188 31444 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:27.562029 1121 raft_consensus.cc:686] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
W20260805 22:24:27.558835 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 49722.8us
I20260805 22:24:27.563879 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:27.564603 31713 raft_consensus.cc:2454] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:27.564841 1095 raft_consensus.cc:2781] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:27.558372 1062 raft_consensus.cc:2781] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:27.565274 1095 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:27.565649 1062 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:27.565735 1095 raft_consensus.cc:3037] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:27.566056 1062 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:27.568967 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:27.569695 31554 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:27.570449 1200 raft_consensus.cc:2781] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:27.570804 1200 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:27.571094 1200 raft_consensus.cc:3037] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:27.571450 31556 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:27.574524 31447 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:27.577504 31492 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:27.579794 31447 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:27.580888 1456 tablet_replica.cc:330] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c: stopping tablet replica
I20260805 22:24:27.583213 1456 raft_consensus.cc:2229] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:27.596472 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:27.594936 1257 raft_consensus.cc:2781] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:27.590741 1381 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:27.564862 1376 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.624882 31554 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:27.607990 1190 raft_consensus.cc:2726] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:27.619367 1381 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:27.622215 1456 raft_consensus.cc:2258] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:27.614363 1257 raft_consensus.cc:686] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:27.574314 1121 consensus_queue.cc:237] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.621003 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:27.688598 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 83137.6us
I20260805 22:24:27.682307 1381 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:27.687618 1291 raft_consensus.cc:2726] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:27.693694 31597 raft_consensus.cc:1261] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:27.709628 1213 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.720562 1467 consensus_queue.cc:1035] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:27.720957 1213 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:27.710960 31603 raft_consensus.cc:1261] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:24:27.709646 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:27.722517 32584 raft_consensus.cc:2379] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 2: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:27.695981 31708 raft_consensus.cc:1261] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:24:27.725073 1467 consensus_queue.cc:1035] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:27.725797 1257 consensus_queue.cc:237] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.733857 1456 ts_tablet_manager.cc:1905] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:27.712550 31705 raft_consensus.cc:1261] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:27.714875 31447 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:27.716109 1457 log.cc:924] Time spent T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20260805 22:24:27.718348 1350 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:27.698578 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:27.756819 1465 raft_consensus.cc:2726] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:27.757853 1263 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:27.760442 31602 raft_consensus.cc:1261] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:24:27.747781 1291 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:27.767537 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:27.728623 1381 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:27.780769 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 173045us
I20260805 22:24:27.769089 1349 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:27.751988 1467 consensus_queue.cc:1035] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.027s
I20260805 22:24:27.770628 31490 raft_consensus.cc:2379] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 2: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
W20260805 22:24:27.805621 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 234235us
I20260805 22:24:27.771703 1366 consensus_queue.cc:1035] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
I20260805 22:24:27.811875 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:27.820178 31501 raft_consensus.cc:3037] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:27.769722 1263 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:27.846625 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 92251.6us
I20260805 22:24:27.849000 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:27.808166 1372 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:27.849669 31602 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:27.780808 1291 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.855772 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:27.858898 1466 consensus_queue.cc:1035] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.081s
I20260805 22:24:27.814549 1376 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:27.795550 1349 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:27.850914 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:27.859433 1291 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:27.820396 1200 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:27.922183 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 73813.1us
I20260805 22:24:27.915033 31596 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:27.923662 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:27.889993 31493 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:27.927893 1270 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:27.928560 1270 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.930239 31705 raft_consensus.cc:1261] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:27.926613 1199 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:27.926556 31493 raft_consensus.cc:595] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:27.944857 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:27.932484 1199 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:27.931579 1349 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:27.862069 1372 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:27.938669 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:27.931198 1121 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:27.937950 1381 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:27.930979 1270 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:27.956643 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:27.954746 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:28.015717 31609 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:24:27.933166 1062 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:27.968175 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:28.020632 31596 raft_consensus.cc:2379] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 3: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
W20260805 22:24:28.018520 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 245584us
I20260805 22:24:28.025458 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:27.953352 32589 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:28.020006 31708 raft_consensus.cc:3037] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:27.960996 1199 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:28.020913 31705 raft_consensus.cc:3037] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:27.958761 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:27.970517 1372 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:24:27.958349 31486 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:27.967016 1200 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:28.037819 31601 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:28.033963 32579 raft_consensus.cc:2379] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 2: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:27.966190 1263 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:28.032642 1456 ts_tablet_manager.cc:1918] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:28.045137 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:27.983542 31501 raft_consensus.cc:2454] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:27.990712 31556 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:28.048269 1475 raft_consensus.cc:2726] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:28.049006 31499 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:24:28.052279 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 61663.7us
I20260805 22:24:27.958442 1140 consensus_queue.cc:1035] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.085s
I20260805 22:24:28.060357 31447 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:28.061496 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:28.061682 1478 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:27.988206 1121 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.062383 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:28.063007 31500 raft_consensus.cc:3037] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:28.061542 1190 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:28.044909 1372 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:28.063840 1190 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.064395 1372 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.066285 1372 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:28.063951 1121 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:27.957641 1095 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:28.062199 31604 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:28.070279 31667 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:28.070943 31602 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:28.068259 31596 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:28.072665 31601 raft_consensus.cc:1261] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260805 22:24:28.057701 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 71168.6us
I20260805 22:24:28.063233 1290 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:28.047017 1257 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:28.076820 31667 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:28.068014 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:28.091622 31501 raft_consensus.cc:1261] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:28.077051 1456 log.cc:1196] T c75e772a0f924080865ddf85ed1e2d86 P 516cb07ce3064ccbbca35e912d62816c: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-1-root/wals/c75e772a0f924080865ddf85ed1e2d86
I20260805 22:24:28.092907 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:28.062273 1478 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.117967 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:28.085633 1349 raft_consensus.cc:2781] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:28.120700 1485 raft_consensus.cc:2781] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:28.121131 1485 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:28.121482 1485 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:28.121874 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:28.121994 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:28.124516 31704 raft_consensus.cc:1261] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:24:28.124682 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260805 22:24:28.089752 1290 raft_consensus.cc:686] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:28.126784 31499 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:28.127820 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:24:28.127326 1290 consensus_queue.cc:237] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:28.128644 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:28.131656 1190 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:28.131877 1478 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:28.084511 1257 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.133095 1095 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:28.089668 1263 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:28.133769 31493 raft_consensus.cc:2396] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 5 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260805 22:24:28.134131 1257 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:28.133886 1263 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.135905 1263 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:28.136708 31493 raft_consensus.cc:1261] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:28.096228 1466 raft_consensus.cc:1050] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994: attempting to promote NON_VOTER 6bf8c694a0ba40978195ae044a917caf to VOTER
I20260805 22:24:28.139293 1466 consensus_queue.cc:237] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 3, Current term: 4, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:28.140712 1485 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.144842 1485 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:28.146096 1062 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:28.148347 32579 raft_consensus.cc:1226] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:28.118477 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:28.141510 1349 raft_consensus.cc:686] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:28.150998 31502 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:28.151464 31667 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:28.096478 31709 raft_consensus.cc:1261] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260805 22:24:28.157998 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 73401.7us
I20260805 22:24:28.162184 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:28.162334 1485 consensus_queue.cc:1035] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:28.163080 32579 raft_consensus.cc:2379] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 2: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:28.163338 31486 raft_consensus.cc:1261] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:28.163980 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:28.164162 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:28.164739 32584 raft_consensus.cc:2354] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:24:28.164995 31706 raft_consensus.cc:2396] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for term 2 because replica has last-logged OpId of term: 1 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 3.
I20260805 22:24:28.165457 1349 consensus_queue.cc:237] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.162201 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:28.172583 31705 raft_consensus.cc:1261] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:28.174885 31708 raft_consensus.cc:2454] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:28.206808 31447 leader_election.cc:398] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Vote denied by peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801) with higher term. Message: Invalid argument: T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:24:28.207316 31447 leader_election.cc:401] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260805 22:24:28.208318 1504 raft_consensus.cc:2726] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801) with higher term. Message: Invalid argument: T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:24:28.212781 1062 consensus_queue.cc:1035] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:28.182647 1257 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:28.214516 1504 raft_consensus.cc:979] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994: : Instructing follower feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:28.215401 1504 raft_consensus.cc:1067] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Signalling peer feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:28.183924 1381 consensus_queue.cc:1046] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260805 22:24:28.216228 31444 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:28.214931 1257 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.220557 1263 consensus_queue.cc:1035] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:28.220201 1504 raft_consensus.cc:3037] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:28.221750 1257 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:28.215318 1495 consensus_queue.cc:1035] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
I20260805 22:24:28.223831 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:28.224622 31600 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:28.181182 1485 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:28.229249 1485 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:28.199591 31500 raft_consensus.cc:2454] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:28.231783 1504 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:28.232064 1485 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:28.232285 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:28.233305 31499 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:28.234536 32584 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEARNER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260805 22:24:28.233073 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:28.237733 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:28.238692 32584 raft_consensus.cc:1226] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:28.238945 1366 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:28.239845 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:28.257583 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 56239.4us
I20260805 22:24:28.258306 31554 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:28.218353 1381 consensus_queue.cc:1225] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer 6bf8c694a0ba40978195ae044a917caf log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260805 22:24:28.262286 31444 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:28.202960 1372 raft_consensus.cc:2726] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:28.267185 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:28.267563 31601 raft_consensus.cc:1261] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:28.270563 31495 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:28.356328 1467 consensus_queue.cc:1035] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20260805 22:24:28.358177 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 91416.6us
I20260805 22:24:28.359680 1140 consensus_queue.cc:1035] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:28.261448 31500 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:28.362921 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:28.363327 1513 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:28.363865 1513 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:28.364296 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:28.364944 31715 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:28.366266 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:28.366842 31601 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:28.367357 1483 consensus_queue.cc:1035] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.104s
I20260805 22:24:28.367543 31339 catalog_manager.cc:4841] TS 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): tablet c75e772a0f924080865ddf85ed1e2d86 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:28.370921 1513 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:28.372051 31667 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:28.255899 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:28.380730 31601 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
W20260805 22:24:28.381994 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 111680us
I20260805 22:24:28.258437 32579 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:28.388413 31667 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:28.391279 31444 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:28.257503 1485 consensus_queue.cc:1046] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.124s
I20260805 22:24:28.273834 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:28.402200 31667 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:28.404088 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:28.406543 31554 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:28.412925 31706 raft_consensus.cc:1226] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:28.414444 31602 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:28.373013 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:28.420265 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:28.421033 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:28.420394 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:28.421495 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:28.421835 31596 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:28.422164 31602 raft_consensus.cc:2354] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:24:28.423204 1475 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:28.424659 1497 raft_consensus.cc:2781] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:28.425344 1484 raft_consensus.cc:2781] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:28.425755 1484 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:28.426090 1484 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:28.426919 31604 raft_consensus.cc:2379] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 3: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:28.424504 31500 raft_consensus.cc:3032] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:28.429428 1486 consensus_queue.cc:1035] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20260805 22:24:28.412537 1131 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:28.415324 1290 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:28.390133 31708 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:28.432817 31707 raft_consensus.cc:1261] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260805 22:24:28.421172 1465 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:28.434410 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:28.436416 31602 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20260805 22:24:28.438321 31601 raft_consensus.cc:2454] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:28.438328 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:28.440739 31709 raft_consensus.cc:2379] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 3: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:28.442382 1517 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:28.442876 1517 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:28.436337 31707 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:56260
I20260805 22:24:28.427245 1350 raft_consensus.cc:2781] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:28.419389 1372 raft_consensus.cc:2726] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:28.420673 31600 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:28.451370 31495 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:28.448995 32529 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:28.430878 31500 raft_consensus.cc:729] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:28.453328 32529 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:28.455395 1510 raft_consensus.cc:2781] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:28.431620 1291 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:28.458544 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:28.435745 1131 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:28.442843 1509 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:28.433554 1497 raft_consensus.cc:686] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:28.439623 1465 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:28.472648 31667 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:28.447408 1350 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:28.481065 1190 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:28.482599 1517 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:28.482985 1291 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:28.483876 32529 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:28.485674 31667 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:28.467331 1376 raft_consensus.cc:2781] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:28.474179 1131 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:28.424842 1475 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.476251 1510 raft_consensus.cc:686] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:28.493625 1350 raft_consensus.cc:3037] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:28.550318 1486 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:28.573652 1213 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:28.574258 1213 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:28.575361 1291 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:28.575960 31596 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:28.576352 1213 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:28.493752 1337 raft_consensus.cc:2781] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:28.577530 1337 raft_consensus.cc:686] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:28.577235 1497 consensus_queue.cc:237] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.578101 1376 raft_consensus.cc:686] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:28.578542 1337 consensus_queue.cc:237] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.579098 1376 consensus_queue.cc:237] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:28.444029 1290 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.490577 1465 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:28.512295 1485 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:28.573764 1486 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:28.570173 1184 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260805 22:24:28.582087 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 91476.7us
I20260805 22:24:28.571045 1510 consensus_queue.cc:237] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:28.574158 32583 raft_consensus.cc:1261] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260805 22:24:28.601051 31554 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:28.608611 1291 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:28.595858 1486 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:28.596153 1184 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:28.612784 1291 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:28.491144 1509 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.620896 1291 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:28.626312 1476 consensus_queue.cc:1035] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260805 22:24:28.598153 1485 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.619797 1184 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:28.639619 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:28.557564 31500 consensus_queue.cc:260] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:28.644883 31501 raft_consensus.cc:2379] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 3: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:28.664470 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:28.665290 31501 raft_consensus.cc:2454] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:28.667619 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:28.668687 31501 raft_consensus.cc:2379] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 2: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:28.639286 31705 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:24:28.643457 1095 consensus_queue.cc:1046] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.023s
I20260805 22:24:28.690059 1484 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.691000 1485 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:28.692134 1484 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:28.693140 1290 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:28.694567 31499 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:28.688776 1466 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:28.696486 1466 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.698382 1466 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:28.690090 1190 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:28.699389 1190 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:28.692159 1475 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:28.687317 1509 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:28.707324 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:28.708611 31501 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:28.717962 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:28.723971 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:28.726186 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 114184us
W20260805 22:24:28.725812 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 129045us
I20260805 22:24:28.747185 31501 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:28.751266 31554 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:28.751380 32529 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:28.722656 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:28.754094 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:28.758402 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:28.761938 31493 raft_consensus.cc:1261] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:28.760202 31713 raft_consensus.cc:2454] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:28.737918 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:28.765493 31494 raft_consensus.cc:2354] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 2. Current term is 3.
I20260805 22:24:28.766729 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:28.770392 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:28.770824 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:28.771397 31608 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:28.748250 1495 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:28.771654 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:28.762472 1140 raft_consensus.cc:2781] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:28.745128 1503 consensus_queue.cc:1035] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.032s
I20260805 22:24:28.761171 1376 raft_consensus.cc:2781] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:28.791698 1349 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:28.792265 1349 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.793852 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:28.794581 1485 consensus_queue.cc:1035] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:28.794586 31715 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:28.794235 1349 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:28.801695 1503 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:28.802301 1503 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.803203 1465 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.804373 1503 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:28.809638 1140 raft_consensus.cc:686] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:28.810689 1140 consensus_queue.cc:237] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.761360 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:28.783252 31594 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260805 22:24:28.786592 32584 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260805 22:24:28.749220 1291 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:28.834095 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 44805.9us
W20260805 22:24:28.860375 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 104775us
I20260805 22:24:28.791860 1376 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:28.772254 1495 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.817274 1537 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:28.875761 31554 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:28.867869 1544 log.cc:924] Time spent T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.078s user 0.002s sys 0.003s
I20260805 22:24:28.877902 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:28.879958 31493 raft_consensus.cc:2379] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 3: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:28.842597 1465 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:28.882354 31709 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
W20260805 22:24:28.888411 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 123490us
I20260805 22:24:28.871800 1376 raft_consensus.cc:3037] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:28.892515 32529 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:28.898926 31444 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:28.905793 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:28.912639 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:28.935873 31492 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:28.899905 1508 raft_consensus.cc:2781] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:28.868557 1350 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:28.899480 31603 raft_consensus.cc:2354] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 2. Current term is 3.
I20260805 22:24:28.936337 31594 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:28.942214 1495 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:28.939010 1508 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:28.980824 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:28.981585 31501 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 3
W20260805 22:24:28.983434 1524 log.cc:924] Time spent T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260805 22:24:28.985941 1486 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:28.986853 1528 log.cc:924] Time spent T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.412s user 0.002s sys 0.000s
I20260805 22:24:28.988541 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:28.985983 1508 raft_consensus.cc:3037] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:28.990093 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:28.990725 31602 raft_consensus.cc:2454] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:28.992214 32589 raft_consensus.cc:1226] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 2. Current term is 3. Ops: [2.5-2.5]
I20260805 22:24:28.994252 31713 raft_consensus.cc:2354] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 2. Current term is 3.
I20260805 22:24:28.999254 31500 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:29.003170 1190 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:29.004851 31494 raft_consensus.cc:1226] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:29.006115 1350 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:28.992138 1486 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:29.009425 1184 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:29.010955 1190 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:28.943477 1540 raft_consensus.cc:2781] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:28.945066 1291 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:28.940573 1480 log.cc:924] Time spent T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.089s user 0.002s sys 0.004s
I20260805 22:24:28.886886 1537 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.950914 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:28.951673 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:28.952431 1131 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:28.895352 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:28.953540 1557 raft_consensus.cc:2726] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:28.901232 1466 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.060s
I20260805 22:24:29.011642 1184 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:29.012058 1508 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:29.007762 31501 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
W20260805 22:24:29.019270 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 106055us
I20260805 22:24:28.938542 31667 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:29.042373 1486 consensus_queue.cc:1046] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:29.055500 31493 raft_consensus.cc:2379] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 3: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:29.054780 31709 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:29.063958 31707 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:29.064790 31492 raft_consensus.cc:1261] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260805 22:24:29.040109 1540 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260805 22:24:29.067654 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 168786us
I20260805 22:24:29.073134 31707 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:29.074121 31444 leader_election.cc:398] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
W20260805 22:24:29.072619 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 134150us
I20260805 22:24:29.075181 31444 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:29.076186 31667 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:29.081143 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:29.081789 31602 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 4
I20260805 22:24:29.083902 31554 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:29.043303 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:29.086908 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:29.087491 31493 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:29.087172 1095 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:29.093811 1095 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:29.045567 1376 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:29.077925 1540 raft_consensus.cc:3037] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:29.083828 1509 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:29.096289 31667 leader_election.cc:398] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Vote denied by peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607) with higher term. Message: Invalid argument: T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 2. Current term is 3.
I20260805 22:24:29.102376 1095 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:29.116330 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:29.116930 31501 raft_consensus.cc:3037] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:29.117529 1566 raft_consensus.cc:2781] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:29.117920 1508 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:29.117933 1566 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:29.118986 1566 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:29.084973 1486 consensus_queue.cc:1225] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer 6bf8c694a0ba40978195ae044a917caf log is divergent from this leader: its last log entry 3.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260805 22:24:29.119940 31493 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:29.106360 1509 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:29.130151 1509 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:29.134569 1131 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:29.104739 31707 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:29.069628 31715 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:29.136817 1537 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:29.137907 1467 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:29.138732 31501 raft_consensus.cc:2454] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:29.107555 31554 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:29.109234 1464 raft_consensus.cc:2781] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:29.140822 1464 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:29.141228 1464 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:29.085059 31490 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:24:29.107103 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:29.139906 31707 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:29.102574 1376 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:29.144364 32589 raft_consensus.cc:2454] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:29.092978 32583 raft_consensus.cc:3037] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:29.145153 1557 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:29.145675 1557 raft_consensus.cc:686] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:29.147148 1557 consensus_queue.cc:237] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:29.107679 31667 leader_election.cc:401] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
W20260805 22:24:29.148532 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 55201.1us
I20260805 22:24:29.149443 1350 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 3
I20260805 22:24:29.151175 31447 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:29.152096 1537 raft_consensus.cc:2781] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:29.152489 1537 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:29.152807 1537 raft_consensus.cc:3037] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:29.155731 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:29.142760 1571 consensus_queue.cc:1035] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:29.160404 1537 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:29.138026 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 62915.7us
I20260805 22:24:29.167017 32529 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:29.167914 32529 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:29.168900 31501 raft_consensus.cc:1226] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:29.170957 31601 raft_consensus.cc:1261] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:29.143276 31554 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:29.171514 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:29.171969 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:29.172200 31597 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:29.172571 31608 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:29.173609 1503 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:29.174002 1503 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:29.174360 1503 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:29.177420 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:29.177983 31601 raft_consensus.cc:3037] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:29.172166 31594 raft_consensus.cc:1261] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260805 22:24:29.166167 31602 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:29.185945 31444 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:29.194393 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:29.195207 31492 raft_consensus.cc:2379] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 3: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:29.198160 31500 raft_consensus.cc:1261] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260805 22:24:29.198537 31500 pending_rounds.cc:76] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994: Aborting all ops after (but not including) 4
I20260805 22:24:29.198776 31500 pending_rounds.cc:97] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260805 22:24:29.199023 31500 raft_consensus.cc:2866] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:24:29.199875 1508 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:29.200436 1508 raft_consensus.cc:686] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:29.201251 1508 consensus_queue.cc:237] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:29.204658 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:29.204309 1540 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:29.198978 1374 raft_consensus.cc:2781] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election won for term 3
W20260805 22:24:29.210387 1062 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260805 22:24:29.211627 31500 raft_consensus.cc:1261] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260805 22:24:29.214413 1553 consensus_queue.cc:1035] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:29.215277 1349 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:29.191712 1095 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.028s
I20260805 22:24:29.220690 1374 raft_consensus.cc:686] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:29.225636 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:29.233013 31609 raft_consensus.cc:1261] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:24:29.227365 1190 raft_consensus.cc:2741] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260805 22:24:29.245180 31494 raft_consensus.cc:3037] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:29.234360 1465 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260805 22:24:29.246555 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:29.247272 31502 raft_consensus.cc:2379] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:29.247617 1476 consensus_queue.cc:1035] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:29.249981 1121 consensus_queue.cc:1035] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:29.252919 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:29.253540 31490 raft_consensus.cc:3032] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:29.253841 31490 raft_consensus.cc:729] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:29.245538 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:29.254621 31490 consensus_queue.cc:260] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:29.255831 31490 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:29.256958 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:29.257769 31492 raft_consensus.cc:2354] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3.
I20260805 22:24:29.258067 31499 raft_consensus.cc:1261] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:24:29.266338 31501 raft_consensus.cc:1261] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260805 22:24:29.267736 31494 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:29.268683 31444 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:29.269742 1465 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:29.270125 1465 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:29.270426 1465 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:29.270828 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:29.271544 31604 raft_consensus.cc:2379] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 2: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:29.260915 1545 consensus_queue.cc:1035] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:24:29.275995 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:29.277616 31608 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:29.235908 1540 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:29.300638 31601 raft_consensus.cc:2454] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:29.303365 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:29.304656 31596 raft_consensus.cc:3037] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:29.307703 31556 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:29.312670 31711 raft_consensus.cc:3032] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:29.313062 31711 raft_consensus.cc:729] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:29.278455 1062 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:29.313730 1062 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:29.313848 31711 consensus_queue.cc:260] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:29.315050 31711 raft_consensus.cc:3037] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:29.317956 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:29.319216 31444 leader_election.cc:398] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 2. Current term is 3.
I20260805 22:24:29.319649 31444 leader_election.cc:401] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260805 22:24:29.323242 31713 raft_consensus.cc:2454] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:29.324872 31707 raft_consensus.cc:2379] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:29.329780 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:29.331269 32529 leader_election.cc:398] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Vote denied by peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607) with higher term. Message: Invalid argument: T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3.
I20260805 22:24:29.331583 32529 leader_election.cc:401] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Cancelling election due to peer responding with higher term
I20260805 22:24:29.332222 1257 raft_consensus.cc:2726] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:29.334455 1489 raft_consensus.cc:2726] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 2. Current term is 3.
I20260805 22:24:29.335796 1372 consensus_queue.cc:1046] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:29.337594 31444 leader_election.cc:398] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 2. Current term is 3.
I20260805 22:24:29.339156 1350 raft_consensus.cc:2726] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607) with higher term. Message: Invalid argument: T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 2. Current term is 3.
I20260805 22:24:29.340073 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:29.341948 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:29.342988 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:29.341243 31667 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:29.223364 1464 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:29.355401 31667 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:29.359792 31494 raft_consensus.cc:1226] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260805 22:24:29.360232 31444 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:29.363163 1374 consensus_queue.cc:237] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:29.363816 31705 raft_consensus.cc:1261] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260805 22:24:29.365618 1372 consensus_queue.cc:1035] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:29.371220 1464 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:29.301417 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:29.377257 31597 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:29.326337 32583 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:29.341094 1503 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:29.380426 1503 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:29.383572 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:29.386961 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:29.387200 31502 raft_consensus.cc:1261] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260805 22:24:29.363263 1562 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.060s
I20260805 22:24:29.386339 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:29.389832 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:29.390341 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:29.390940 31713 raft_consensus.cc:2454] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:29.396695 31667 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:29.399696 31501 raft_consensus.cc:2379] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 3: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:29.315828 1062 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:29.241497 1349 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:24:29.357676 1517 raft_consensus.cc:3037] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:29.418449 31444 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:29.416654 31711 raft_consensus.cc:1261] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 5. (term mismatch)
I20260805 22:24:29.419834 31711 pending_rounds.cc:76] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385: Aborting all ops after (but not including) 4
I20260805 22:24:29.420161 31711 pending_rounds.cc:97] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385: Aborting uncommitted NO_OP operation due to leader change: 2.5
I20260805 22:24:29.420517 31711 raft_consensus.cc:2866] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:24:29.357513 1350 raft_consensus.cc:2726] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:29.424355 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:29.380460 1566 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:29.400594 1561 raft_consensus.cc:2726] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:29.425587 1537 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:29.360771 1048 consensus_queue.cc:1035] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.074s
I20260805 22:24:29.360512 1213 consensus_queue.cc:1035] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.074s
I20260805 22:24:29.381074 1121 raft_consensus.cc:2781] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:29.431756 31667 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
W20260805 22:24:29.613781 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 182104us
I20260805 22:24:29.448568 31596 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:29.615267 31608 raft_consensus.cc:1226] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260805 22:24:29.565958 31707 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:29.617754 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:29.642452 31711 raft_consensus.cc:3037] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:29.642630 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:29.402873 31486 raft_consensus.cc:1261] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
W20260805 22:24:29.425462 1022 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 3 current term: 3 is higher.
I20260805 22:24:29.506776 31356 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47946 (request call id 182) took 3598 ms. Trace:
I20260805 22:24:29.651773 31357 catalog_manager.cc:5495] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:29.643349 31596 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:29.662356 31357 catalog_manager.cc:5495] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:29.428354 1131 raft_consensus.cc:2781] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:29.669577 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:29.425288 1349 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:29.432022 1562 raft_consensus.cc:2726] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:29.397449 1465 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:29.407409 1484 consensus_queue.cc:1035] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.266s
I20260805 22:24:29.670172 31714 raft_consensus.cc:2354] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4.
I20260805 22:24:29.654196 31490 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
W20260805 22:24:29.688504 1509 consensus_meta.cc:287] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.558s user 0.008s sys 0.001s
I20260805 22:24:29.688794 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:29.639727 1559 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:29.692989 31711 raft_consensus.cc:2454] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
W20260805 22:24:29.676862 1557 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.513s user 0.016s sys 0.053s
I20260805 22:24:29.720479 31495 raft_consensus.cc:2354] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4.
I20260805 22:24:29.681092 1121 raft_consensus.cc:686] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:29.722188 1121 consensus_queue.cc:237] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:29.725904 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 98287.6us
W20260805 22:24:29.729676 1515 log.cc:924] Time spent T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260805 22:24:29.732298 31667 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:24:29.744634 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 129866us
I20260805 22:24:29.716903 1200 consensus_queue.cc:1035] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.035s
W20260805 22:24:29.753080 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 110102us
W20260805 22:24:29.757732 31706 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.610s user 0.026s sys 0.077s
W20260805 22:24:29.758137 31706 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.610s user 0.026s sys 0.077s
W20260805 22:24:29.719970 1557 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s user 0.016s sys 0.053s
W20260805 22:24:29.730825 1568 log.cc:924] Time spent T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260805 22:24:29.766861 31444 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:29.655678 31356 rpcz_store.cc:276] 0805 22:24:25.927583 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:25.950950 (+ 23367us) service_pool.cc:227] Handling call
0805 22:24:29.506699 (+3555749us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0805 22:24:28.020546 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0805 22:24:28.036977 (+ 16431us) write_op.cc:432] Acquiring schema lock in shared mode
0805 22:24:28.037016 (+ 39us) write_op.cc:435] Acquired schema lock
0805 22:24:28.037028 (+ 12us) tablet.cc:660] Decoding operations
0805 22:24:28.037574 (+ 546us) write_op.cc:620] Acquiring the partition lock for write op
0805 22:24:28.037629 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0805 22:24:28.037657 (+ 28us) tablet.cc:683] Acquiring locks for 3 operations
0805 22:24:28.037989 (+ 332us) tablet.cc:699] Row locks acquired
0805 22:24:28.038005 (+ 16us) write_op.cc:260] PREPARE: finished
0805 22:24:28.038144 (+ 139us) write_op.cc:270] Start()
0805 22:24:28.038254 (+ 110us) write_op.cc:276] Timestamp: P: 1785968668038100 usec, L: 0
0805 22:24:28.038273 (+ 19us) op_driver.cc:331] REPLICATION: starting
0805 22:24:28.077688 (+ 39415us) log.cc:842] Serialized 1365 byte log entry
0805 22:24:28.423633 (+345945us) op_driver.cc:444] REPLICATION: finished
0805 22:24:28.673173 (+249540us) write_op.cc:301] APPLY: starting
0805 22:24:28.676841 (+ 3668us) tablet.cc:1365] starting BulkCheckPresence
0805 22:24:28.676987 (+ 146us) tablet.cc:1368] finished BulkCheckPresence
0805 22:24:28.677143 (+ 156us) tablet.cc:1370] starting ApplyRowOperation cycle
0805 22:24:28.682138 (+ 4995us) tablet.cc:1381] finished ApplyRowOperation cycle
0805 22:24:28.682319 (+ 181us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=3
0805 22:24:28.682375 (+ 56us) write_op.cc:312] APPLY: finished
0805 22:24:28.682571 (+ 196us) log.cc:842] Serialized 35 byte log entry
0805 22:24:28.682720 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:28.682809 (+ 89us) write_op.cc:454] Released schema lock
0805 22:24:28.682849 (+ 40us) write_op.cc:341] FINISH: Updating metrics
0805 22:24:28.683129 (+ 280us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:28.683156 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":53108,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":209927,"apply.run_cpu_time_us":2539,"apply.run_wall_time_us":15720,"num_ops":3,"prepare.queue_time_us":311722,"prepare.run_cpu_time_us":3480,"prepare.run_wall_time_us":72170,"replication_time_us":376507,"thread_start_us":196813,"threads_started":2}]]}
I20260805 22:24:29.703362 1131 raft_consensus.cc:686] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:29.743628 1559 raft_consensus.cc:686] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:29.794802 1559 consensus_queue.cc:237] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:29.799414 31499 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.538s user 0.025s sys 0.027s
W20260805 22:24:29.800485 31499 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.539s user 0.025s sys 0.027s
I20260805 22:24:29.717722 1140 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
W20260805 22:24:29.776163 1528 log.cc:924] Time spent T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.057s user 0.001s sys 0.000s
I20260805 22:24:29.814128 31444 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:29.717314 1509 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:29.813820 1540 raft_consensus.cc:2781] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:29.818377 1562 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:29.849221 31709 raft_consensus.cc:2354] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 2. Current term is 3.
I20260805 22:24:29.818498 1561 raft_consensus.cc:2781] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:29.857030 31554 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:29.861766 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:29.862447 31596 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:24:29.856773 1376 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.622s user 0.029s sys 0.050s
W20260805 22:24:29.863692 1376 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.629s user 0.029s sys 0.050s
I20260805 22:24:29.809005 1140 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
W20260805 22:24:29.878013 31602 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.601s user 0.004s sys 0.011s
W20260805 22:24:29.879243 31602 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.602s user 0.004s sys 0.011s
I20260805 22:24:29.851202 1540 raft_consensus.cc:686] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:29.859252 1561 raft_consensus.cc:686] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:29.869910 1566 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:24:29.922601 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 154121us
I20260805 22:24:29.853542 1562 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:29.931147 1509 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:29.934324 31596 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:29.919961 1465 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:29.942723 31444 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:29.962829 1465 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election won for term 4
W20260805 22:24:29.963194 1517 consensus_meta.cc:287] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.530s user 0.012s sys 0.000s
I20260805 22:24:29.946511 1140 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:29.946406 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:29.964138 31596 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:29.948313 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:29.881498 1257 consensus_queue.cc:1035] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260805 22:24:29.966439 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:29.967396 31707 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:24:29.967963 31493 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.007s sys 0.015s
W20260805 22:24:29.968736 31493 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.700s user 0.007s sys 0.015s
I20260805 22:24:29.957724 1540 consensus_queue.cc:237] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:29.963662 1517 raft_consensus.cc:2726] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election lost for term 2. Reason: Vote denied by peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607) with higher term. Message: Invalid argument: T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3.
I20260805 22:24:29.965734 31604 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:29.922166 1550 consensus_queue.cc:1046] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.222s
I20260805 22:24:29.937924 1557 raft_consensus.cc:2741] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election decision vote started in defunct term 3: won
W20260805 22:24:29.996681 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 127982us
I20260805 22:24:30.013777 31707 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:30.032485 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:30.025979 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 51401.6us
I20260805 22:24:29.963342 1465 raft_consensus.cc:686] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:30.036397 31596 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
W20260805 22:24:30.099823 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 59958.4us
W20260805 22:24:30.156625 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 57479.3us
I20260805 22:24:29.928077 1131 consensus_queue.cc:237] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:30.027961 1350 raft_consensus.cc:2781] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:30.251807 1140 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:30.038499 1599 consensus_queue.cc:1035] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.316s
W20260805 22:24:30.444360 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 480395us
W20260805 22:24:30.471742 1592 log.cc:924] Time spent T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.544s user 0.000s sys 0.001s
I20260805 22:24:30.473703 31499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1463) took 1213 ms. Trace:
I20260805 22:24:30.474080 31499 rpcz_store.cc:276] 0805 22:24:29.259848 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:29.259947 (+ 99us) service_pool.cc:227] Handling call
0805 22:24:29.260223 (+ 276us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:29.260541 (+ 318us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:29.260566 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:29.260969 (+ 403us) log.cc:842] Serialized 26 byte log entry
0805 22:24:29.801247 (+540278us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:24:29.818190 (+ 16943us) log.cc:842] Serialized 14 byte log entry
0805 22:24:29.855206 (+ 37016us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:29.855283 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:30.473476 (+618193us) raft_consensus.cc:1671] finished
0805 22:24:30.473501 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:30.473642 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":32740,"prepare.queue_time_us":16860,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":32949,"raft.run_cpu_time_us":91,"raft.run_wall_time_us":92,"spinlock_wait_cycles":8822656,"thread_start_us":539902,"threads_started":1,"wal-append.queue_time_us":619078}
I20260805 22:24:30.477720 1465 consensus_queue.cc:237] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:30.478256 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:30.510202 1597 log.cc:924] Time spent T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.537s user 0.001s sys 0.000s
I20260805 22:24:30.512243 31603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 949) took 1115 ms. Trace:
I20260805 22:24:30.512652 31603 rpcz_store.cc:276] 0805 22:24:29.396360 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:29.396473 (+ 113us) service_pool.cc:227] Handling call
0805 22:24:29.403512 (+ 7039us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:29.426392 (+ 22880us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:29.426427 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:29.426886 (+ 459us) log.cc:842] Serialized 26 byte log entry
0805 22:24:29.818242 (+391356us) raft_consensus.cc:1631] Marking committed up to 1
0805 22:24:29.878174 (+ 59932us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:29.878849 (+ 675us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:30.511943 (+633094us) raft_consensus.cc:1671] finished
0805 22:24:30.511967 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:30.512146 (+ 179us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":142851,"raft.queue_time_us":15917,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":39569,"thread_start_us":390961,"threads_started":1,"wal-append.queue_time_us":492683}
W20260805 22:24:30.526345 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 49172.6us
W20260805 22:24:30.598423 31667 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.630s user 0.037s sys 0.031s
W20260805 22:24:30.607357 31667 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.641s user 0.038s sys 0.031s
W20260805 22:24:30.607712 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 641005us
I20260805 22:24:30.281093 1561 consensus_queue.cc:237] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:30.643113 1203 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:30.649577 1379 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:30.650456 1379 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:30.648034 1203 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:30.432108 1140 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:30.467254 31486 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.511s user 0.010s sys 0.020s
W20260805 22:24:30.679749 32587 consensus_meta.cc:287] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.697s user 0.006s sys 0.003s
I20260805 22:24:30.680294 32587 raft_consensus.cc:2454] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:30.715811 31713 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:43958
I20260805 22:24:30.716583 31713 raft_consensus.cc:595] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:30.716955 31713 raft_consensus.cc:603] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:24:30.734915 1379 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:30.735610 1623 log.cc:924] Time spent T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.304s user 0.000s sys 0.001s
W20260805 22:24:30.738338 1607 log.cc:924] Time spent T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.380s user 0.001s sys 0.000s
I20260805 22:24:30.739781 31602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56384 (request call id 198) took 1464 ms. Trace:
I20260805 22:24:30.740154 31602 rpcz_store.cc:276] 0805 22:24:29.275287 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:29.275400 (+ 113us) service_pool.cc:227] Handling call
0805 22:24:29.275724 (+ 324us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:29.276070 (+ 346us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:29.276095 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:29.276502 (+ 407us) log.cc:842] Serialized 26 byte log entry
0805 22:24:29.879581 (+603079us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:24:29.879618 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:29.879682 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:30.739555 (+859873us) raft_consensus.cc:1671] finished
0805 22:24:30.739575 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:30.739709 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1141,"raft.run_cpu_time_us":147,"raft.run_wall_time_us":16703,"spinlock_wait_cycles":6400,"thread_start_us":602742,"threads_started":1,"wal-append.queue_time_us":747170}
I20260805 22:24:30.741605 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260805 22:24:30.743489 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:24:30.744220 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:30.746172 32586 raft_consensus.cc:1261] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260805 22:24:30.672421 1626 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:30.779688 1562 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:30.800947 1257 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:30.802917 1626 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:30.974716 1616 log.cc:924] Time spent T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.164s user 0.000s sys 0.001s
I20260805 22:24:30.975440 1140 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:31.000865 1626 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:31.001932 1626 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:31.002564 1626 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:31.017952 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 309328us
W20260805 22:24:31.019134 31604 consensus_meta.cc:287] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 1.046s user 0.010s sys 0.000s
I20260805 22:24:31.019618 31604 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:31.022871 31704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 1977) took 1407 ms. Trace:
I20260805 22:24:31.030190 31604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59030 (request call id 955) took 1072 ms. Trace:
W20260805 22:24:31.025434 31554 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.741s user 0.000s sys 0.010s
W20260805 22:24:31.047667 31554 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.763s user 0.000s sys 0.010s
W20260805 22:24:31.048035 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 890598us
I20260805 22:24:31.030718 31604 rpcz_store.cc:276] 0805 22:24:29.947905 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:29.948061 (+ 156us) service_pool.cc:227] Handling call
0805 22:24:31.020150 (+1072089us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:24:31.023324 31704 rpcz_store.cc:276] 0805 22:24:29.615746 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:29.615856 (+ 110us) service_pool.cc:227] Handling call
0805 22:24:29.652602 (+ 36746us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:29.652980 (+ 378us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:29.653006 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:29.756019 (+103013us) spinlock_profiling.cc:243] Waited 9.19 ms on lock 0x7b2c00004908. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272fa4be11 00007f272fa5b442 00007f27310a3c11 00007f273106858a 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772
0805 22:24:29.756877 (+ 858us) log.cc:842] Serialized 564 byte log entry
0805 22:24:30.002149 (+245272us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:24:30.002192 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:30.002291 (+ 99us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:31.011830 (+1009539us) raft_consensus.cc:1671] finished
0805 22:24:31.011847 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:31.022782 (+ 10935us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21141888,"thread_start_us":244920,"threads_started":1,"wal-append.queue_time_us":501214}
W20260805 22:24:31.028205 1550 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.768s user 0.009s sys 0.018s
I20260805 22:24:31.016218 1140 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:30.893900 1257 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:31.058597 1550 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.805s user 0.009s sys 0.019s
I20260805 22:24:31.059885 1379 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:31.067601 1626 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:31.064903 1517 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:31.076512 1639 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:31.076350 1476 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:31.063483 1631 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:31.079068 1509 consensus_queue.cc:1046] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.005s
I20260805 22:24:31.075731 31486 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40740 (request call id 961) took 1314 ms. Trace:
I20260805 22:24:31.077450 1372 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:31.070568 1131 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:31.095664 1372 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:31.099123 1498 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:31.099730 1498 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:31.100267 1372 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:31.076162 1379 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:31.103271 1379 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:31.103855 1291 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:31.104413 1291 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:31.106441 1291 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:31.106912 1291 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:31.107331 31711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1533) took 1075 ms. Trace:
I20260805 22:24:31.107542 1559 raft_consensus.cc:3032] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:31.108433 31711 rpcz_store.cc:276] 0805 22:24:30.031428 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:30.031542 (+ 114us) service_pool.cc:227] Handling call
0805 22:24:30.257588 (+226046us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:30.265736 (+ 8148us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:30.265767 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:30.284202 (+ 18435us) log.cc:842] Serialized 26 byte log entry
0805 22:24:30.874636 (+590434us) raft_consensus.cc:1631] Marking committed up to 1
0805 22:24:30.874676 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:30.874743 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:31.107138 (+232395us) raft_consensus.cc:1671] finished
0805 22:24:31.107157 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:31.107276 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7721,"raft.queue_time_us":223179,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":31212,"thread_start_us":440763,"threads_started":1,"wal-append.queue_time_us":790510}
I20260805 22:24:31.071782 1140 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:31.121490 31667 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:31.091594 1636 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:31.107997 1559 raft_consensus.cc:729] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
W20260805 22:24:31.132977 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 84298.5us
I20260805 22:24:31.096040 1626 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:31.133991 1337 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:31.145300 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:31.145085 1337 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:31.093758 1476 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:31.113818 1291 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:31.152266 1291 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:31.092803 1517 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:31.156488 31554 leader_election.cc:398] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4.
I20260805 22:24:31.187739 31554 leader_election.cc:401] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260805 22:24:31.140228 1376 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:31.109113 1627 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:31.124370 1550 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:31.097513 1131 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:31.094110 1631 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:31.195007 1517 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:31.202245 31667 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:31.154526 1653 raft_consensus.cc:2781] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:31.195757 1291 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:31.214502 1291 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:31.217046 31669 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.689s user 0.016s sys 0.027s
I20260805 22:24:31.218360 1337 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:24:31.221413 1584 log.cc:924] Time spent T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 1.555s user 0.032s sys 0.052s
W20260805 22:24:31.222822 31669 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.821s user 0.018s sys 0.027s
I20260805 22:24:31.198608 1257 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:31.159245 1626 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:31.254369 1657 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:31.095547 1639 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:31.262288 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 70564.4us
W20260805 22:24:31.264849 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 108511us
I20260805 22:24:31.273532 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:31.198972 1376 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:31.261605 1291 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:31.233477 1653 raft_consensus.cc:686] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:31.140559 1636 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:31.226883 1476 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:31.233206 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:31.250195 1257 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:31.255498 1657 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:31.269435 1654 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:31.204645 1550 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:31.296306 1655 raft_consensus.cc:2726] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4.
I20260805 22:24:31.297642 1655 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:31.297065 1659 consensus_queue.cc:1035] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.022s
I20260805 22:24:31.298130 1140 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:31.212435 1559 consensus_queue.cc:260] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:31.200917 1627 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:31.314334 1643 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:31.315469 1643 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:31.316066 1627 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:31.319110 1654 raft_consensus.cc:686] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:31.322897 1376 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:31.324554 1609 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:31.340598 1658 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:31.326803 1498 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:31.350746 1498 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:31.351759 1498 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:31.290405 1257 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:31.355808 1631 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:31.356218 1649 log.cc:924] Time spent T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.183s user 0.000s sys 0.002s
I20260805 22:24:31.354295 1653 consensus_queue.cc:237] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:31.327173 1559 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:31.370071 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:31.341842 1658 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:31.373939 1659 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:31.377660 1659 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:31.383898 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:31.379644 1550 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:31.355850 1257 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:31.393443 1626 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:31.395659 1639 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:31.396637 1639 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:31.400624 1657 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:31.403067 1636 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:31.404386 1669 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:31.298544 1655 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:31.426364 1658 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:31.325464 1609 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:31.321069 1654 consensus_queue.cc:237] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:31.409778 1257 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:31.418327 1643 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:31.407508 1639 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:31.405493 1669 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:31.433663 1257 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:31.448058 1659 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:31.469704 1669 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:24:31.466223 1639 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:31.474159 1655 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:32.198529 1559 consensus_meta.cc:287] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.832s user 0.015s sys 0.000s
I20260805 22:24:31.578423 1609 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:32.316395 1257 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:32.393150 1498 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:32.721712 1653 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.258s user 0.009s sys 0.006s
W20260805 22:24:32.725307 1653 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.261s user 0.009s sys 0.006s
W20260805 22:24:32.731756 1654 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.094s user 0.027s sys 0.046s
W20260805 22:24:32.733247 1654 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.096s user 0.027s sys 0.046s
W20260805 22:24:33.677395 32586 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.233s user 0.013s sys 0.020s
W20260805 22:24:33.677934 32586 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.233s user 0.013s sys 0.020s
I20260805 22:24:33.860379 1131 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:33.998272 1121 raft_consensus.cc:3032] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:33.999038 1121 raft_consensus.cc:729] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:34.000029 1121 consensus_queue.cc:260] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.035681 1203 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:34.239768 1589 log.cc:924] Time spent T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 4.319s user 0.035s sys 0.028s
I20260805 22:24:34.251037 1131 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:34.251677 1131 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.253938 1131 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:34.267184 1121 raft_consensus.cc:3037] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:34.357266 31708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58348 (request call id 257) took 4481 ms. Trace:
W20260805 22:24:34.358217 1621 log.cc:924] Time spent T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 3.555s user 0.022s sys 0.027s
W20260805 22:24:34.384778 1690 log.cc:924] Time spent T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 1.655s user 0.007s sys 0.020s
I20260805 22:24:34.432287 31708 rpcz_store.cc:276] 0805 22:24:29.876137 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:29.880266 (+ 4129us) service_pool.cc:227] Handling call
0805 22:24:34.356304 (+4476038us) spinlock_profiling.cc:243] Waited 4.54 ms on lock 0x7b58002d0698. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:24:34.356599 (+ 295us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:34.356819 (+ 220us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:34.356850 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:34.356897 (+ 47us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:24:34.356933 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:34.357001 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:34.357147 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10446592}
I20260805 22:24:34.445848 1545 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:34.446553 1545 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.447727 32579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57372 (request call id 473) took 4153 ms. Trace:
I20260805 22:24:34.449698 32579 rpcz_store.cc:276] 0805 22:24:30.294609 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:30.357569 (+ 62960us) service_pool.cc:227] Handling call
0805 22:24:34.446886 (+4089317us) spinlock_profiling.cc:243] Waited 10.9 ms on lock 0x7b5800005198. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:24:34.447106 (+ 220us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:34.447329 (+ 223us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:34.447355 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:34.447400 (+ 45us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:24:34.447431 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:34.447494 (+ 63us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:34.447638 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":25004928}
W20260805 22:24:34.448889 32529 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.261s user 0.014s sys 0.034s
W20260805 22:24:34.451309 32529 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.264s user 0.014s sys 0.034s
W20260805 22:24:34.451637 1619 log.cc:924] Time spent T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 3.600s user 0.004s sys 0.029s
I20260805 22:24:34.452319 1545 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:34.451696 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 3.26408e+06us
I20260805 22:24:34.465251 1485 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:34.465600 1131 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:34.466176 1558 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:34.465981 1485 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.466637 1558 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.468318 1485 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.468782 1131 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.471246 1131 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:34.472081 31709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56260 (request call id 1971) took 5175 ms (client timeout 1550 ms). Trace:
I20260805 22:24:34.472517 1131 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:34.473011 1131 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.475278 1131 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:34.468523 1558 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:34.477253 1694 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:34.477340 1558 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:34.477933 1694 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.478018 1558 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:34.472550 31709 rpcz_store.cc:269] 0805 22:24:29.297047 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:29.341656 (+ 44609us) service_pool.cc:227] Handling call
0805 22:24:34.471809 (+5130153us) spinlock_profiling.cc:243] Waited 502 ms on lock 0x7b54000ed154. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:34.471972 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1155071232}
I20260805 22:24:34.480130 1349 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:34.480751 1349 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.480177 1558 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.482234 1694 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.482815 1349 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:24:34.483350 31556 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.276s user 0.023s sys 0.062s
W20260805 22:24:34.483827 31556 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.276s user 0.023s sys 0.062s
I20260805 22:24:34.484624 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.486181 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.488961 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.490260 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.491128 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.491724 31711 raft_consensus.cc:2454] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 1.
I20260805 22:24:34.493081 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.494598 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.494946 1464 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:34.495478 1464 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.496042 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.497404 1464 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.497898 1464 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:34.498327 1464 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.500152 1464 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:34.502331 31556 connection.cc:232] RPC call timeout handler was delayed by -1.3878428561147302s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21860000000000002s for a response
I20260805 22:24:34.502434 1464 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:34.503029 1464 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.505239 1464 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:34.506237 31669 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.224s user 0.020s sys 0.035s
W20260805 22:24:34.506703 31669 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.225s user 0.021s sys 0.035s
I20260805 22:24:34.508250 1121 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:31.090768 31486 rpcz_store.cc:276] 0805 22:24:29.760903 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:29.764469 (+ 3566us) service_pool.cc:227] Handling call
0805 22:24:29.955015 (+190546us) spinlock_profiling.cc:243] Waited 3.28 ms on lock 0x7b5800001898. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:24:29.955205 (+ 190us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:29.955542 (+ 337us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:29.955575 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:29.956019 (+ 444us) log.cc:842] Serialized 26 byte log entry
0805 22:24:30.830792 (+874773us) raft_consensus.cc:1631] Marking committed up to 1
0805 22:24:30.878829 (+ 48037us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:31.046893 (+168064us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:31.050155 (+ 3262us) raft_consensus.cc:1671] finished
0805 22:24:31.050170 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:31.073320 (+ 23150us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1047,"raft.run_cpu_time_us":112,"raft.run_wall_time_us":112,"spinlock_wait_cycles":7553920,"thread_start_us":716006,"threads_started":1,"wal-append.queue_time_us":491742}
I20260805 22:24:34.510021 1553 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:34.510515 1553 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.508736 1121 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.512992 1121 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:34.513511 1121 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:34.514005 1121 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.515839 1695 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:34.515951 1121 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:34.516384 1695 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.518625 1545 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:34.518936 1695 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.519564 1545 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:34.520416 31554 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.236s user 0.013s sys 0.029s
I20260805 22:24:34.521835 1553 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.522234 1545 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:34.523016 1545 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:34.525472 1694 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:34.526070 1694 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:34.525185 32529 connection.cc:232] RPC call timeout handler was delayed by -1.7284318071324378s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1905s for a response
W20260805 22:24:34.526609 31554 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.242s user 0.014s sys 0.030s
W20260805 22:24:34.527359 32529 connection.cc:232] RPC call timeout handler was delayed by -1.7284318071324378s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1905s for a response
W20260805 22:24:34.527407 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 3.26025e+06us
I20260805 22:24:34.528594 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.531384 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.529459 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.533576 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.534157 32585 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:24:34.535015 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.529714 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.542997 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:34.532559 31559 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 3.242s user 0.045s sys 0.061s
W20260805 22:24:34.543815 31559 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 3.253s user 0.045s sys 0.061s
I20260805 22:24:34.546307 32529 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:34.546802 1484 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:34.547164 1656 raft_consensus.cc:2781] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:34.547626 1656 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:34.547596 1557 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:34.548622 1557 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.549912 31608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 1928) took 4715 ms. Trace:
I20260805 22:24:34.550587 1557 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:34.551028 31556 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:34.530172 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.547374 1484 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.552670 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.548059 1656 raft_consensus.cc:3037] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:34.550310 31608 rpcz_store.cc:276] 0805 22:24:29.834331 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:29.834773 (+ 442us) service_pool.cc:227] Handling call
0805 22:24:29.849608 (+ 14835us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:29.927743 (+ 78135us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:29.927772 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:29.936467 (+ 8695us) log.cc:842] Serialized 564 byte log entry
0805 22:24:30.285253 (+348786us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:24:30.285288 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:34.549508 (+4264220us) spinlock_profiling.cc:243] Waited 77.2 ms on lock 0x7b54000069d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:34.549558 (+ 50us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:34.549601 (+ 43us) raft_consensus.cc:1671] finished
0805 22:24:34.549612 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:34.549830 (+ 218us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":177581312,"thread_start_us":348518,"threads_started":1,"wal-append.queue_time_us":706615,"wal-append.run_cpu_time_us":2862,"wal-append.run_wall_time_us":1412472}
I20260805 22:24:34.547796 1661 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:34.555572 1661 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.556514 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.553567 1484 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:34.557868 1484 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:34.558503 1634 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:34.558403 1484 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.559041 1634 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.560223 31596 raft_consensus.cc:1261] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:24:34.560156 1661 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.561141 1661 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:34.551103 1557 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:34.561262 1634 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.566320 1634 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
W20260805 22:24:34.567550 32587 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57382 (request call id 501) took 5101 ms (client timeout 2837 ms). Trace:
W20260805 22:24:34.568943 32587 rpcz_store.cc:269] 0805 22:24:29.465917 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:29.466578 (+ 661us) service_pool.cc:227] Handling call
0805 22:24:34.567356 (+5100778us) spinlock_profiling.cc:243] Waited 270 ms on lock 0x7b54004a33d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:34.567465 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":621119744}
I20260805 22:24:34.561337 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.570335 1634 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.570698 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.571341 31604 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 1.
I20260805 22:24:34.571862 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.573560 1634 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:34.574075 1634 raft_consensus.cc:2781] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:34.574409 1634 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:34.574774 1634 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:34.575735 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.576416 31603 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:34.565646 1557 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.564442 31493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52098 (request call id 178) took 5304 ms. Trace:
I20260805 22:24:34.578357 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.579104 31604 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 1.
I20260805 22:24:34.579360 1557 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:34.578423 31493 rpcz_store.cc:276] 0805 22:24:29.259555 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:29.263179 (+ 3624us) service_pool.cc:227] Handling call
0805 22:24:29.263485 (+ 306us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:29.267387 (+ 3902us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:29.267420 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:29.267836 (+ 416us) log.cc:842] Serialized 26 byte log entry
0805 22:24:29.969348 (+701512us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:24:30.028231 (+ 58883us) log.cc:842] Serialized 14 byte log entry
0805 22:24:34.563847 (+4535616us) spinlock_profiling.cc:243] Waited 7.77 ms on lock 0x7b180036db40. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e0b74ca 00007f2737798f99 00007f272df7b990 00007f272e13d864 00007f272e13d155 00007f27310603d0 00007f2731072576 00007f27310830d7 00007f2731083058
0805 22:24:34.564033 (+ 186us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:34.564106 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:34.564147 (+ 41us) raft_consensus.cc:1671] finished
0805 22:24:34.564246 (+ 99us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:34.564351 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5736,"prepare.queue_time_us":56976,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":52,"raft.queue_time_us":3635,"raft.run_cpu_time_us":80,"raft.run_wall_time_us":81,"spinlock_wait_cycles":17866496,"thread_start_us":700989,"threads_started":1,"wal-append.queue_time_us":1082792,"wal-append.run_cpu_time_us":2919,"wal-append.run_wall_time_us":1295329}
I20260805 22:24:34.523815 1545 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.583613 1545 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:34.583097 1656 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.585013 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.587116 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.588057 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.588691 31603 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 1.
I20260805 22:24:34.589370 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.591174 1350 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:34.591625 1350 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:34.594010 1656 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:34.594983 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.595808 1661 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.596261 32584 raft_consensus.cc:2454] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:24:34.532485 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.598704 32579 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:34.599200 1634 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.600435 31608 raft_consensus.cc:2379] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 2: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:34.601195 31706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1522) took 5455 ms. Trace:
I20260805 22:24:34.609612 31706 rpcz_store.cc:276] 0805 22:24:29.145456 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:29.146209 (+ 753us) service_pool.cc:227] Handling call
0805 22:24:29.147100 (+ 891us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:29.147303 (+ 203us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:29.147325 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:29.147759 (+ 434us) log.cc:842] Serialized 26 byte log entry
0805 22:24:29.758477 (+610718us) raft_consensus.cc:1631] Marking committed up to 1
0805 22:24:29.758520 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:29.758599 (+ 79us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:34.239040 (+4480441us) spinlock_profiling.cc:243] Waited 2.57 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:24:34.600889 (+361849us) raft_consensus.cc:1671] finished
0805 22:24:34.600911 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:34.601135 (+ 224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5899520,"thread_start_us":610382,"threads_started":1,"wal-append.queue_time_us":700605}
I20260805 22:24:34.602499 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.604883 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.606841 31556 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:34.612185 31357 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34400 (request call id 283) took 7410 ms. Trace:
I20260805 22:24:34.612998 31556 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:34.613731 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.607888 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.613884 1557 raft_consensus.cc:2781] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:34.614477 1557 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:34.614854 1557 raft_consensus.cc:3037] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:34.615377 32583 raft_consensus.cc:1226] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:34.617004 1661 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:34.617663 1661 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:34.618558 1661 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:34.535627 32585 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:34.621119 1484 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:34.623500 31351 catalog_manager.cc:5495] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 1 to 3, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 3 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:34.619767 1661 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:34.629887 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.630194 1599 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:34.630352 1545 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:34.630709 1599 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.631176 1545 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.632162 31351 catalog_manager.cc:5495] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 1 to 2, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 2 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:34.632958 1599 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:34.634595 1545 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:34.634101 31351 catalog_manager.cc:5495] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 3 to 4, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 4 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:24:34.634888 1557 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.636188 31351 catalog_manager.cc:5495] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:34.639535 1200 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:34.639999 1200 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.641933 1200 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.642395 1200 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:34.642935 31603 raft_consensus.cc:1226] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:34.642805 1200 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.643553 1661 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.644713 1200 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.646656 31715 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:56246 (request call id 807) took 5331 ms. Trace:
I20260805 22:24:34.647014 31715 rpcz_store.cc:276] 0805 22:24:29.315086 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:29.369066 (+ 53980us) service_pool.cc:227] Handling call
0805 22:24:34.646039 (+5276973us) spinlock_profiling.cc:243] Waited 623 ms on lock 0x7b5400205654. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fedaf0 00007f2731070f2a 00007f2735fadc6e 00007f272f6a4807 00007f272f6a4792 00007f272f6a46c1 00007f272f6a463c 00007f272f6a38f2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:34.646574 (+ 535us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1432435456}
I20260805 22:24:34.612542 31357 rpcz_store.cc:276] 0805 22:24:27.201479 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:27.201798 (+ 319us) service_pool.cc:227] Handling call
0805 22:24:34.612086 (+7410288us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0805 22:24:30.358419 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0805 22:24:30.789483 (+431064us) write_op.cc:432] Acquiring schema lock in shared mode
0805 22:24:30.798880 (+ 9397us) write_op.cc:435] Acquired schema lock
0805 22:24:30.798906 (+ 26us) tablet.cc:660] Decoding operations
0805 22:24:30.815687 (+ 16781us) write_op.cc:620] Acquiring the partition lock for write op
0805 22:24:30.815778 (+ 91us) write_op.cc:641] Partition lock acquired for write op
0805 22:24:30.815867 (+ 89us) tablet.cc:683] Acquiring locks for 2 operations
0805 22:24:30.897198 (+ 81331us) tablet.cc:699] Row locks acquired
0805 22:24:30.897223 (+ 25us) write_op.cc:260] PREPARE: finished
0805 22:24:30.937623 (+ 40400us) write_op.cc:270] Start()
0805 22:24:30.966448 (+ 28825us) write_op.cc:276] Timestamp: P: 1785968670936696 usec, L: 0
0805 22:24:30.966506 (+ 58us) op_driver.cc:331] REPLICATION: starting
0805 22:24:30.998370 (+ 31864us) log.cc:842] Serialized 929 byte log entry
0805 22:24:31.255760 (+257390us) op_driver.cc:444] REPLICATION: finished
0805 22:24:31.334283 (+ 78523us) write_op.cc:301] APPLY: starting
0805 22:24:31.359064 (+ 24781us) tablet.cc:1365] starting BulkCheckPresence
0805 22:24:31.388358 (+ 29294us) tablet.cc:1368] finished BulkCheckPresence
0805 22:24:31.419825 (+ 31467us) tablet.cc:1370] starting ApplyRowOperation cycle
0805 22:24:31.432510 (+ 12685us) tablet.cc:1381] finished ApplyRowOperation cycle
0805 22:24:31.446059 (+ 13549us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0805 22:24:31.447829 (+ 1770us) write_op.cc:312] APPLY: finished
0805 22:24:31.454858 (+ 7029us) log.cc:842] Serialized 29 byte log entry
0805 22:24:31.460822 (+ 5964us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:31.462571 (+ 1749us) write_op.cc:454] Released schema lock
0805 22:24:31.462656 (+ 85us) write_op.cc:341] FINISH: Updating metrics
0805 22:24:31.464275 (+ 1619us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:31.464333 (+ 58us) write_op.cc:454] Released schema lock
0805 22:24:33.821623 (+2357290us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x7b5400002dd4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731064198 00007f2734a77ade 00007f2734a772b8 00007f2734a7bba1 00007f2734a7bb59 00007f2734a7bae9 00007f2734a7bab1 00007f2734a7adad 00007f27373a4204 00007f27373a4039
Metrics: {"mutex_wait_us":7682,"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":47145,"apply.run_cpu_time_us":4476,"apply.run_wall_time_us":155852,"num_ops":2,"prepare.queue_time_us":580657,"prepare.run_cpu_time_us":31271,"prepare.run_wall_time_us":3536614,"raft.queue_time_us":156224,"raft.run_cpu_time_us":5137,"raft.run_wall_time_us":17560,"replication_time_us":286607,"spinlock_wait_cycles":61668736,"thread_start_us":401385,"threads_started":4,"wal-append.queue_time_us":65995,"wal-append.run_cpu_time_us":43698,"wal-append.run_wall_time_us":1487149}]]}
I20260805 22:24:34.645458 1557 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.650007 32586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57366 (request call id 299) took 3287 ms. Trace:
I20260805 22:24:34.650383 32586 rpcz_store.cc:276] 0805 22:24:31.362189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:31.362423 (+ 234us) service_pool.cc:227] Handling call
0805 22:24:31.363812 (+ 1389us) raft_consensus.cc:1457] Updating replica for 2 ops
0805 22:24:31.441551 (+ 77739us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:31.441621 (+ 70us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0805 22:24:31.443844 (+ 2223us) log.cc:842] Serialized 527 byte log entry
0805 22:24:33.678446 (+2234602us) raft_consensus.cc:1631] Marking committed up to 3
0805 22:24:33.678497 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:34.384043 (+705546us) spinlock_profiling.cc:243] Waited 1.47 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:24:34.649681 (+265638us) spinlock_profiling.cc:243] Waited 71.7 ms on lock 0x7b54005cf654. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:34.649737 (+ 56us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:34.649789 (+ 52us) raft_consensus.cc:1671] finished
0805 22:24:34.649800 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:34.649949 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":390,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":167,"spinlock_wait_cycles":168255488,"thread_start_us":2233845,"threads_started":1,"wal-append.queue_time_us":1285799}
I20260805 22:24:34.657367 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:34.660189 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.660823 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.661355 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.661890 1661 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.662199 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.662968 31604 raft_consensus.cc:2396] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for term 5 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260805 22:24:34.664057 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.664707 31715 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:34.666651 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:34.667181 32591 raft_consensus.cc:3037] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:34.686683 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.687486 31713 raft_consensus.cc:2454] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:34.688870 1350 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.690454 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.691716 32591 raft_consensus.cc:2454] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:34.692121 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.720252 31713 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:34.720427 32583 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:34.692687 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.721412 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.722115 31713 raft_consensus.cc:2454] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 1.
I20260805 22:24:34.725003 1576 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:34.725126 1350 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.725505 1190 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:34.693190 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.694489 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:34.695678 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.697093 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:34.697129 1553 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:34.699426 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.524679 1558 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:34.701797 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:31.429032 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 226865us
I20260805 22:24:34.707211 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.741243 31711 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:34.709772 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.710790 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.710952 1694 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:34.712467 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:34.746552 31714 raft_consensus.cc:3037] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:34.713163 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.747211 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.747501 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.741281 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:34.748546 31706 raft_consensus.cc:2396] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260805 22:24:34.748538 32591 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:34.747946 31604 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
W20260805 22:24:34.699512 32529 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.905s (SENT) [suppressed 12 similar messages]
I20260805 22:24:34.726074 1467 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:34.725083 1562 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:34.725273 1489 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:34.726272 1466 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:34.725700 1576 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.758380 1467 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.758567 1466 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.760658 1290 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:34.760519 1489 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.761545 1478 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:34.762108 1478 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.756893 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.750677 1464 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:34.764031 1540 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:34.760533 1537 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:34.758000 1381 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:34.764605 1540 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.766265 1537 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.765687 1464 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.612218 1634 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.742393 1553 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.737874 1465 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:34.730038 1514 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:34.770363 1465 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.770694 1514 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.772684 1551 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:34.765012 1381 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.773200 1551 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.775249 1504 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:34.775365 1464 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:34.786346 31714 raft_consensus.cc:2454] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:34.787323 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.739818 32579 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:34.795061 31711 raft_consensus.cc:2379] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:34.795102 31596 raft_consensus.cc:1226] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 1. Current term is 2. Ops: []
W20260805 22:24:34.797188 31444 connection.cc:232] RPC call timeout handler was delayed by -4.0013100209180266s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15500000000000003s for a response
I20260805 22:24:34.797281 32591 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:34.799304 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.799810 31596 raft_consensus.cc:2454] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:24:34.750730 31708 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:34.751047 1503 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:24:34.806015 31554 connection.cc:232] RPC call timeout handler was delayed by -3.4480917321052402s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.197s for a response
I20260805 22:24:34.805743 1503 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.805068 1485 raft_consensus.cc:2932] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
W20260805 22:24:34.806393 31554 connection.cc:232] RPC call timeout handler was delayed by -3.0742968879640102s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1665s for a response
I20260805 22:24:34.806784 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.807420 31711 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:34.807015 31554 leader_election.cc:398] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Vote denied by peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607) with higher term. Message: Invalid argument: T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4.
I20260805 22:24:34.808310 1703 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:34.808813 1703 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.751374 1558 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.810169 31707 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:34.811149 1558 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:34.730286 1190 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.829108 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.797953 1062 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:34.796767 1291 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:34.775863 1504 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.842224 32581 raft_consensus.cc:2932] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:34.843241 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.843832 31596 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:34.844811 1504 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.845317 1504 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:34.845685 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.845762 1504 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.761226 1290 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.841953 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.825526 1702 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:34.848810 1540 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.849082 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.850088 1489 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:34.850238 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.825800 1703 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:34.850718 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.875792 31706 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:34.851099 1467 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:34.852406 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.853972 1381 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:34.854290 1291 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.856305 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.857527 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.786684 1562 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:34.875243 31714 raft_consensus.cc:2454] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:24:34.879163 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.887985 1562 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:34.884027 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.884369 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:34.884804 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.892338 31604 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 1.
I20260805 22:24:34.854796 1551 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:34.840694 1095 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:34.855249 1466 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.900905 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.902905 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:34.903570 31706 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:24:34.906715 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:24:34.916337 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 103423us
W20260805 22:24:34.917286 31444 connection.cc:232] RPC call timeout handler was delayed by -2.8372496669180691s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.20870000000000002s for a response
I20260805 22:24:34.917467 31554 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:34.918639 31554 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:34.922916 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.923058 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.923352 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.924059 31596 raft_consensus.cc:1261] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260805 22:24:34.924021 31594 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:34.874291 31708 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
W20260805 22:24:34.925776 31667 connection.cc:232] RPC call timeout handler was delayed by -3.2243577360641211s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21120000000000003s for a response
I20260805 22:24:34.918666 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.928845 31602 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:34.855911 1062 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:34.929569 31667 connection.cc:232] RPC call timeout handler was delayed by -3.1355417540762573s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1847s for a response
I20260805 22:24:34.930482 31554 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:34.932672 31667 connection.cc:232] RPC call timeout handler was delayed by -2.8916417539585382s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2118s for a response
I20260805 22:24:34.933559 31554 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:34.933954 1537 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:34.934607 1661 raft_consensus.cc:2726] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:34.935071 1485 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:56246 (request call id 801) took 9235 ms. Trace:
I20260805 22:24:34.935554 1485 rpcz_store.cc:276] 0805 22:24:25.699244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:25.701894 (+ 2650us) service_pool.cc:227] Handling call
0805 22:24:25.792626 (+ 90732us) spinlock_profiling.cc:243] Waited 18.7 ms on lock 0x7b58001fd3b0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2730fe3f63 00007f2731047aba 00007f2731063a7a 00007f2731064ada 00007f2731063c8b 00007f273106f084 00007f273106e743 00007f2731066c1a 00007f2735faa6d9 00007f272f694184
0805 22:24:25.977544 (+184918us) log.cc:842] Serialized 501 byte log entry
0805 22:24:34.934856 (+8957312us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4637,"raft.queue_time_us":59925,"raft.run_cpu_time_us":2402,"raft.run_wall_time_us":187414,"spinlock_wait_cycles":43124864,"thread_start_us":372230,"threads_started":2,"wal-append.queue_time_us":256327,"wal-append.run_cpu_time_us":4506,"wal-append.run_wall_time_us":1338216}
I20260805 22:24:34.938910 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet e6cf90ae3fc840ceb48be8c790be97d9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:34.938818 1576 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.940244 1478 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:34.942553 31594 raft_consensus.cc:1261] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260805 22:24:34.944999 31447 connection.cc:232] RPC call timeout handler was delayed by -2.6763154549989849s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1935s for a response
W20260805 22:24:34.945416 31447 connection.cc:232] RPC call timeout handler was delayed by -1.7300330658908933s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2837s for a response
I20260805 22:24:34.895311 31705 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
W20260805 22:24:34.932636 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 233136us
I20260805 22:24:34.947896 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:34.934659 31554 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:34.951089 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.951690 31554 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:34.953084 31556 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:34.962608 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:34.968437 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:34.953783 31667 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:34.982144 1465 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:35.023931 1698 log.cc:924] Time spent T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.167s user 0.000s sys 0.000s
I20260805 22:24:34.948210 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.030372 1504 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:34.953019 1561 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:34.982218 1604 raft_consensus.cc:2781] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:34.986977 1661 raft_consensus.cc:2726] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:34.902486 1095 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:34.951035 1696 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:35.032608 1062 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:35.031553 1576 raft_consensus.cc:2932] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:34.885197 1702 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:35.034441 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 45559.2us
I20260805 22:24:34.946638 1634 raft_consensus.cc:2726] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:35.035761 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:35.040711 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:35.040984 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 87264us
I20260805 22:24:34.952425 1466 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:34.954548 1540 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:34.975818 1574 raft_consensus.cc:2726] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:35.033234 1712 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
W20260805 22:24:35.045231 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 94084.9us
I20260805 22:24:34.987651 1489 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:35.076191 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:35.076134 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 125763us
I20260805 22:24:35.081437 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.034464 1381 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:35.034901 1703 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:35.084942 31604 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:35.074748 1553 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:35.075979 1604 raft_consensus.cc:686] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:35.051121 1131 raft_consensus.cc:2781] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:35.134399 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.061165 1514 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:35.131282 31444 leader_election.cc:398] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 2. Current term is 3.
I20260805 22:24:35.086190 31667 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:35.135774 31554 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:35.050611 1475 raft_consensus.cc:2781] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:35.148412 1475 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:35.148818 1475 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
W20260805 22:24:35.149482 1690 log.cc:924] Time spent T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.060s user 0.000s sys 0.000s
I20260805 22:24:35.151597 1095 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:35.155956 1711 log.cc:924] Time spent T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
W20260805 22:24:35.146741 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 60635.5us
I20260805 22:24:35.146150 31444 leader_election.cc:401] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260805 22:24:35.086361 1503 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:35.158759 31493 raft_consensus.cc:2354] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3.
I20260805 22:24:35.163524 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.164217 31501 raft_consensus.cc:2379] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 3: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
W20260805 22:24:35.141232 1714 log.cc:924] Time spent T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.108s user 0.000s sys 0.001s
W20260805 22:24:35.142341 1708 log.cc:924] Time spent T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.105s user 0.000s sys 0.000s
I20260805 22:24:35.181563 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.178872 1615 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:35.184654 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.185493 1615 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:35.165179 31554 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:35.165758 31667 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:35.168579 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.162461 1190 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:35.200790 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.078220 1561 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:35.173353 1290 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:35.174832 1604 consensus_queue.cc:237] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:35.173133 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.177004 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.186429 1615 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:35.146864 1703 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:35.195820 31490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:52098 (request call id 180) took 5320 ms. Trace:
I20260805 22:24:35.197798 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.193835 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.199843 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.179862 1514 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 3
I20260805 22:24:35.173133 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.157477 1566 raft_consensus.cc:2726] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:35.143705 1379 consensus_queue.cc:1035] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260805 22:24:35.162915 1291 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:35.217139 1190 consensus_queue.cc:1046] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260805 22:24:35.230217 1190 raft_consensus.cc:3032] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:35.230538 1190 raft_consensus.cc:729] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:35.231513 1190 consensus_queue.cc:260] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:35.232648 1190 raft_consensus.cc:3037] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:35.093112 1466 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:35.236794 1466 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:35.128038 1540 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:35.161280 1489 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:35.238601 1540 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:35.239734 1489 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:35.201550 1350 raft_consensus.cc:2781] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:35.240744 1350 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:35.241349 1350 raft_consensus.cc:3037] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:35.188048 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.179890 1694 consensus_queue.cc:1046] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.025s
I20260805 22:24:35.206900 31667 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:35.147544 1131 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:35.188978 31604 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:35.247778 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.207839 31601 raft_consensus.cc:1226] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:35.206084 31447 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:35.211807 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:24:35.268371 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 62405.3us
I20260805 22:24:35.214407 31603 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:35.216670 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.219264 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.221750 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:35.223227 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 58131.5us
I20260805 22:24:35.144639 1712 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:35.223811 1503 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:35.303253 1562 raft_consensus.cc:2726] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:35.227054 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.241644 1561 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:35.320648 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.230482 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.321545 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.153221 1702 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:35.146937 1381 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:35.224742 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 137929us
I20260805 22:24:35.406399 32529 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:35.407794 32529 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:35.408289 1566 raft_consensus.cc:3032] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:35.408691 1566 raft_consensus.cc:729] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:35.267432 31594 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:35.212391 32579 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:35.269608 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.410009 31604 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:24:35.241756 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.410848 31597 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:35.247004 31667 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:35.239770 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:24:35.411623 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 164690us
I20260805 22:24:35.411854 31600 raft_consensus.cc:2379] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:35.272424 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.412706 31501 raft_consensus.cc:2454] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:35.412889 1615 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:35.413656 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:35.413928 31667 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:35.414265 1699 raft_consensus.cc:2741] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260805 22:24:35.414695 1349 raft_consensus.cc:2781] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:35.415102 1349 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:35.415268 31667 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:35.415673 1349 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:35.415983 1553 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:35.408821 32529 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:35.412762 1558 raft_consensus.cc:2781] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:35.416889 1558 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:35.417166 1558 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:35.270478 31554 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:35.317154 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.417474 1379 raft_consensus.cc:2781] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:35.417965 1379 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:35.418309 1379 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:35.269584 31500 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:35.320192 1503 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:35.421615 1503 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:35.422081 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.422879 1349 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:35.423317 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.424476 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.425030 31500 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:35.226919 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.421170 32529 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:35.227082 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.425802 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.425941 1349 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:35.416131 31667 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:35.240141 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:35.300616 31444 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:35.426041 1699 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
W20260805 22:24:35.427044 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 155218us
I20260805 22:24:35.333416 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.427709 31495 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:35.426584 1604 raft_consensus.cc:2726] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:35.429885 1566 consensus_queue.cc:260] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:35.322992 31596 raft_consensus.cc:1261] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260805 22:24:35.427351 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.394873 31492 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:35.431363 1566 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:35.431509 31494 raft_consensus.cc:2396] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 6 because replica has last-logged OpId of term: 3 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260805 22:24:35.222661 31490 rpcz_store.cc:276] 0805 22:24:29.875212 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:34.522462 (+4647250us) service_pool.cc:227] Handling call
0805 22:24:34.522868 (+ 406us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:34.710465 (+187597us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:34.710501 (+ 36us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:34.711011 (+ 510us) log.cc:842] Serialized 26 byte log entry
0805 22:24:34.789447 (+ 78436us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:24:34.796964 (+ 7517us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:34.814827 (+ 17863us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:35.195581 (+380754us) raft_consensus.cc:1671] finished
0805 22:24:35.195607 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:35.195735 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":183698,"raft.queue_time_us":185733,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":136,"thread_start_us":40130,"threads_started":1,"wal-append.queue_time_us":191092}
I20260805 22:24:35.435258 31501 raft_consensus.cc:2454] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:35.396456 1712 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:35.319979 1121 raft_consensus.cc:2726] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260805 22:24:35.397070 1732 log.cc:924] Time spent T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.094s user 0.002s sys 0.004s
I20260805 22:24:35.447880 1503 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:35.397823 1475 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:35.452884 1475 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:35.399394 31603 raft_consensus.cc:1226] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:35.408630 1545 raft_consensus.cc:2781] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:35.455238 1545 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:35.455665 1545 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:35.455847 1703 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:35.318411 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 49424.2us
I20260805 22:24:35.424695 1615 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:35.426460 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.457175 31502 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:35.320884 31609 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:35.323714 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:35.430342 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 159894us
I20260805 22:24:35.458706 31447 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:35.460182 31554 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:35.266122 1131 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:35.433332 31500 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:35.460407 1562 raft_consensus.cc:2726] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:35.461443 1562 consensus_queue.cc:1046] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:35.434617 31667 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:35.434706 32529 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:35.396232 31608 raft_consensus.cc:1261] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:35.435748 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.435833 31444 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:35.436323 1694 consensus_queue.cc:1035] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:35.463389 1562 raft_consensus.cc:3032] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:35.465587 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.466758 31486 raft_consensus.cc:2454] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:35.468148 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:35.440296 1558 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:35.441481 1190 consensus_queue.cc:1046] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.008s
I20260805 22:24:35.469792 1694 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:35.470567 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.464630 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:35.470693 1121 consensus_queue.cc:1035] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:35.471452 31667 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:35.472353 1553 raft_consensus.cc:2726] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:35.472528 31499 raft_consensus.cc:1261] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:24:35.472976 31667 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:35.470155 1190 raft_consensus.cc:3032] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:35.470412 1694 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:35.471416 31493 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:35.475054 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.476424 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.437651 1489 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:35.477557 1489 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:35.478322 31609 raft_consensus.cc:1261] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260805 22:24:35.465615 1562 raft_consensus.cc:729] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:35.398797 1514 raft_consensus.cc:2726] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 2. Current term is 3.
I20260805 22:24:35.464694 32529 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:35.479533 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:35.479888 1374 raft_consensus.cc:2781] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:35.480464 1374 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:35.480850 32529 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:35.481189 1374 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:35.484454 31603 raft_consensus.cc:1261] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:35.465828 1733 raft_consensus.cc:2781] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:35.485610 1733 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:35.485747 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.486228 1733 raft_consensus.cc:3037] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:35.474144 1190 raft_consensus.cc:729] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:35.474251 1694 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:35.474313 31499 raft_consensus.cc:1261] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 3. (term mismatch)
I20260805 22:24:35.489679 31499 pending_rounds.cc:76] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994: Aborting all ops after (but not including) 2
I20260805 22:24:35.477988 1489 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
W20260805 22:24:35.487896 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 52145.1us
I20260805 22:24:35.490746 31444 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:35.441684 1349 raft_consensus.cc:2741] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260805 22:24:35.488679 31493 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:35.440948 1379 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:35.472268 31500 raft_consensus.cc:1261] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260805 22:24:35.488435 1712 raft_consensus.cc:2781] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:35.489931 31499 pending_rounds.cc:97] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260805 22:24:35.493268 31499 raft_consensus.cc:2866] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:24:35.495862 31502 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:35.449899 1699 raft_consensus.cc:2726] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:35.478893 1558 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:35.489974 1562 consensus_queue.cc:260] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:35.498799 31554 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:35.490604 1656 raft_consensus.cc:2781] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:35.503222 31444 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:35.506904 1140 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:35.507509 1140 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:35.507972 1140 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:35.508782 1379 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:35.509474 1374 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:35.512506 1562 raft_consensus.cc:3037] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:35.513463 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.512756 1374 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:35.516551 1350 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:35.517768 31554 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:35.518719 1291 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:35.519120 1291 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:35.519457 1291 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:35.489854 1190 consensus_queue.cc:260] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:35.522751 1190 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:35.524261 1545 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:35.502626 1615 raft_consensus.cc:2781] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:35.526520 1615 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:35.526887 1615 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
W20260805 22:24:35.529727 32627 debug-util.cc:398] Leaking SignalData structure 0x7b080025e800 after lost signal to thread 29786
I20260805 22:24:35.530773 1576 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:35.531245 1576 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:35.503700 1723 consensus_queue.cc:1035] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260805 22:24:35.531678 1576 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:35.501797 1372 raft_consensus.cc:2781] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:35.530946 1503 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:35.532483 32627 debug-util.cc:398] Leaking SignalData structure 0x7b0800244740 after lost signal to thread 31402
I20260805 22:24:35.528409 1561 consensus_queue.cc:1046] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 4.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.005s
I20260805 22:24:35.533890 1553 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:35.534580 1553 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:35.535725 1702 consensus_queue.cc:1035] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:35.536999 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.504983 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.538908 1656 raft_consensus.cc:686] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:35.540251 1656 consensus_queue.cc:237] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:35.529588 1514 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:35.543001 1514 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:35.543188 1561 raft_consensus.cc:3032] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:35.543756 1561 raft_consensus.cc:729] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:35.545158 1561 consensus_queue.cc:260] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:35.547540 1561 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:35.555866 1350 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:35.497435 1712 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:35.556752 1712 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:35.543612 1514 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:35.558209 31554 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:35.559568 31554 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:35.560070 1475 raft_consensus.cc:2781] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:35.560392 1604 raft_consensus.cc:2781] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:35.560671 1475 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:35.561072 1475 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:35.561024 1604 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:35.562036 1604 raft_consensus.cc:3037] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:35.564373 31554 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:35.576646 1503 raft_consensus.cc:2726] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:35.578678 1553 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:35.578490 31600 consensus_queue.cc:237] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:35.579327 32627 debug-util.cc:398] Leaking SignalData structure 0x7b08000fa8e0 after lost signal to thread 31535
W20260805 22:24:35.579900 32627 debug-util.cc:398] Leaking SignalData structure 0x7b0800252940 after lost signal to thread 31536
I20260805 22:24:35.582185 31554 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:35.571588 31357 catalog_manager.cc:5495] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:35.588138 31357 catalog_manager.cc:5495] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 3, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 3 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:35.589903 31357 catalog_manager.cc:5495] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 3, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 3 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:35.607870 31554 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:35.609993 31554 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:35.610880 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.611670 31357 catalog_manager.cc:5495] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 3, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 3 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:35.607820 31351 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34394 (request call id 294) took 8075 ms. Trace:
W20260805 22:24:35.619963 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 143086us
I20260805 22:24:35.621917 31357 catalog_manager.cc:5495] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:35.623617 31357 catalog_manager.cc:5495] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 2 to 3, leader changed from 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:35.626075 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.625577 31357 catalog_manager.cc:5495] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 3, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 3 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:35.628687 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.630295 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:35.631199 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:35.632529 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.633296 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.634420 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.635111 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.639002 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:35.639272 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.639475 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.641160 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.642973 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.643834 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.644543 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.644958 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.646085 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.646216 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.647364 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.649570 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.581279 1372 raft_consensus.cc:686] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:35.560334 1545 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:35.655537 31715 raft_consensus.cc:1226] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:35.655966 31500 raft_consensus.cc:1226] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 2. Current term is 3. Ops: []
W20260805 22:24:35.657752 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 165753us
W20260805 22:24:35.648677 32627 debug-util.cc:398] Leaking SignalData structure 0x7b0800362600 after lost signal to thread 31643
I20260805 22:24:35.610560 1610 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:35.620234 1503 raft_consensus.cc:2781] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:35.650175 1493 raft_consensus.cc:2781] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:35.695215 1140 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:35.697576 31603 raft_consensus.cc:1226] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260805 22:24:35.698454 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.648442 1634 raft_consensus.cc:2781] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:35.696341 1131 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:35.664206 32627 debug-util.cc:398] Leaking SignalData structure 0x7b08002d5b20 after lost signal to thread 31646
I20260805 22:24:35.702414 1372 consensus_queue.cc:237] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.5, Last appended by leader: 5, Current term: 5, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:35.703613 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.705708 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.706287 31708 raft_consensus.cc:1261] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260805 22:24:35.687556 1610 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:35.720860 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.707048 1381 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:35.666880 1503 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:35.687866 1493 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:35.706755 1634 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:35.776515 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.706665 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.778851 31500 raft_consensus.cc:1261] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:35.778548 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.781967 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.772018 1503 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:35.772684 1493 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:35.697106 31705 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:35.792677 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.800691 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.802166 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:35.803476 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:35.796401 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.806509 31502 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:35.798384 1131 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:35.799222 1604 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:35.809122 1604 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:35.809635 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:35.804384 31444 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:35.780186 1634 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:35.814433 31444 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:35.816263 31444 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:35.817801 31444 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:35.819075 31444 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:35.811357 31667 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:35.812681 31708 raft_consensus.cc:1226] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:35.771152 1712 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:35.833351 32627 debug-util.cc:398] Leaking SignalData structure 0x7b0800242ce0 after lost signal to thread 31748
I20260805 22:24:35.843937 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:35.857613 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.857818 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:35.859184 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:35.860288 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:35.886359 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:35.887427 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:35.828636 1703 raft_consensus.cc:2726] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:35.828874 1562 raft_consensus.cc:2726] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:35.828217 1540 raft_consensus.cc:2726] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:35.823438 1381 raft_consensus.cc:2726] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:35.821817 1131 raft_consensus.cc:2726] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:35.861363 1478 raft_consensus.cc:2726] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:35.861334 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:35.907601 32579 raft_consensus.cc:2454] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:24:35.925509 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:35.926476 31602 raft_consensus.cc:2379] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 2: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:35.835081 1291 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:35.941226 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.842518 1553 consensus_queue.cc:1035] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.068s
I20260805 22:24:35.843001 1350 consensus_queue.cc:1035] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.074s
I20260805 22:24:35.944259 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:24:35.944749 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 165549us
I20260805 22:24:35.954594 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:35.955116 31602 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:35.956174 31601 raft_consensus.cc:1226] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
W20260805 22:24:35.943756 32627 debug-util.cc:398] Leaking SignalData structure 0x7b08002abc00 after lost signal to thread 32633
I20260805 22:24:35.963544 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:35.961952 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.966102 31714 raft_consensus.cc:1226] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:35.969058 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:35.969703 31499 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:35.618781 31351 rpcz_store.cc:276] 0805 22:24:27.531944 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:27.532079 (+ 135us) service_pool.cc:227] Handling call
0805 22:24:35.607731 (+8075652us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0805 22:24:34.668996 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0805 22:24:34.670617 (+ 1621us) write_op.cc:432] Acquiring schema lock in shared mode
0805 22:24:34.670650 (+ 33us) write_op.cc:435] Acquired schema lock
0805 22:24:34.670663 (+ 13us) tablet.cc:660] Decoding operations
0805 22:24:34.671255 (+ 592us) write_op.cc:620] Acquiring the partition lock for write op
0805 22:24:34.671343 (+ 88us) write_op.cc:641] Partition lock acquired for write op
0805 22:24:34.671367 (+ 24us) tablet.cc:683] Acquiring locks for 4 operations
0805 22:24:34.671754 (+ 387us) tablet.cc:699] Row locks acquired
0805 22:24:34.671772 (+ 18us) write_op.cc:260] PREPARE: finished
0805 22:24:34.671899 (+ 127us) write_op.cc:270] Start()
0805 22:24:34.675965 (+ 4066us) write_op.cc:276] Timestamp: P: 1785968674671862 usec, L: 0
0805 22:24:34.675994 (+ 29us) op_driver.cc:331] REPLICATION: starting
0805 22:24:34.768775 (+ 92781us) log.cc:842] Serialized 1759 byte log entry
0805 22:24:35.230938 (+462163us) op_driver.cc:444] REPLICATION: finished
0805 22:24:35.402204 (+171266us) write_op.cc:301] APPLY: starting
0805 22:24:35.402306 (+ 102us) tablet.cc:1365] starting BulkCheckPresence
0805 22:24:35.402455 (+ 149us) tablet.cc:1368] finished BulkCheckPresence
0805 22:24:35.441409 (+ 38954us) tablet.cc:1370] starting ApplyRowOperation cycle
0805 22:24:35.451164 (+ 9755us) tablet.cc:1381] finished ApplyRowOperation cycle
0805 22:24:35.451577 (+ 413us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0805 22:24:35.451678 (+ 101us) write_op.cc:312] APPLY: finished
0805 22:24:35.452091 (+ 413us) log.cc:842] Serialized 41 byte log entry
0805 22:24:35.454083 (+ 1992us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:35.499138 (+ 45055us) write_op.cc:454] Released schema lock
0805 22:24:35.501071 (+ 1933us) write_op.cc:341] FINISH: Updating metrics
0805 22:24:35.581991 (+ 80920us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:35.594717 (+ 12726us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":8,"child_traces":[["op",{"apply.queue_time_us":171015,"apply.run_cpu_time_us":3966,"apply.run_wall_time_us":217958,"num_ops":4,"prepare.queue_time_us":4689,"prepare.run_cpu_time_us":35685,"prepare.run_wall_time_us":222325,"raft.queue_time_us":73826,"raft.run_cpu_time_us":17899,"raft.run_wall_time_us":41512,"replication_time_us":554843,"thread_start_us":212703,"threads_started":4,"wal-append.queue_time_us":266777}]]}
W20260805 22:24:35.988137 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 169129us
I20260805 22:24:35.988929 31444 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:35.944599 1140 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:35.994390 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 156309us
I20260805 22:24:35.945860 1733 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:36.003257 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 164673us
I20260805 22:24:36.007177 31444 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.010671 31554 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:36.015061 31602 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
W20260805 22:24:36.019968 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 70178.4us
I20260805 22:24:36.019757 1503 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:36.020754 31556 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:36.022029 31556 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:36.022245 1503 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:36.024792 31444 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:35.988654 1610 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:36.026795 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.025349 1503 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:36.031723 31444 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.007784 1190 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:35.969583 1489 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:36.088632 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 58429.9us
I20260805 22:24:36.101531 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.025722 1478 raft_consensus.cc:2726] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:36.101166 1610 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:36.080508 1540 raft_consensus.cc:2726] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:36.106240 31444 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.107957 31444 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.108908 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.012626 1514 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:36.109558 31502 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
W20260805 22:24:36.104156 1576 consensus_meta.cc:287] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.572s user 0.012s sys 0.007s
W20260805 22:24:36.107652 1744 log.cc:924] Time spent T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.264s user 0.001s sys 0.000s
I20260805 22:24:36.138528 1514 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:36.109253 1703 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:36.139276 31502 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:36.139535 1489 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:36.140694 1291 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:36.106787 1290 consensus_queue.cc:1035] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20260805 22:24:36.131572 1478 raft_consensus.cc:2726] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:36.132467 31715 raft_consensus.cc:1226] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:36.113225 1610 raft_consensus.cc:686] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
W20260805 22:24:36.134757 1694 consensus_meta.cc:287] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.645s user 0.014s sys 0.002s
W20260805 22:24:36.135924 1566 consensus_meta.cc:287] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.700s user 0.011s sys 0.006s
I20260805 22:24:36.107502 1768 raft_consensus.cc:2781] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:24:36.144143 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.162778 31499 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:36.145282 31502 raft_consensus.cc:595] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.104614 1763 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:36.162778 31609 raft_consensus.cc:1226] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
W20260805 22:24:36.167418 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 145452us
W20260805 22:24:36.167660 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 103650us
I20260805 22:24:36.164920 1768 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:36.168370 1290 consensus_queue.cc:1046] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:36.144992 1576 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:36.168696 1694 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:36.169512 1290 raft_consensus.cc:3032] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:36.169853 1290 raft_consensus.cc:729] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:36.171705 31499 raft_consensus.cc:1226] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:36.105418 1190 consensus_queue.cc:1046] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.048s
I20260805 22:24:36.168505 1610 consensus_queue.cc:237] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:36.177739 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 1
I20260805 22:24:36.178094 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.178915 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.170540 1576 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:36.162699 1381 consensus_queue.cc:1046] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.019s
W20260805 22:24:36.165423 1762 log.cc:924] Time spent T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.067s user 0.001s sys 0.000s
I20260805 22:24:36.179368 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:24:36.186877 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:36.180742 31500 raft_consensus.cc:1261] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260805 22:24:36.179925 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.184150 31707 raft_consensus.cc:1226] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:36.180071 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.177790 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.185865 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.204470 31502 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:36.206471 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:36.211835 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:24:36.181777 1475 consensus_meta.cc:287] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.620s user 0.017s sys 0.002s
I20260805 22:24:36.215875 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.182626 1712 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:36.216689 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:36.217981 32591 raft_consensus.cc:1261] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:36.218503 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.183969 1694 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:36.173877 1768 raft_consensus.cc:3037] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
W20260805 22:24:36.202958 1615 consensus_meta.cc:287] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.668s user 0.008s sys 0.004s
I20260805 22:24:36.219632 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.222343 31705 raft_consensus.cc:1226] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:36.187695 1503 raft_consensus.cc:686] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:36.225065 1634 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:36.182560 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.227262 32579 raft_consensus.cc:1261] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:36.240497 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:36.241569 1203 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:36.241786 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:36.242118 1203 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:36.242926 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:36.244056 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:36.243817 1503 consensus_queue.cc:237] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:36.219913 1475 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:36.247013 1475 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:36.215247 1290 consensus_queue.cc:260] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:36.248734 1634 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:36.231983 31500 raft_consensus.cc:2362] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3. Re-sending same reply.
I20260805 22:24:36.218315 1763 consensus_queue.cc:1046] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.027s
I20260805 22:24:36.250790 1763 raft_consensus.cc:3032] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:36.251121 1763 raft_consensus.cc:729] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:36.251830 1763 consensus_queue.cc:260] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:36.245144 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:36.254311 31600 raft_consensus.cc:1226] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:36.249555 31604 raft_consensus.cc:1226] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:36.254513 1748 consensus_queue.cc:1035] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:36.255620 1634 consensus_queue.cc:1035] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:36.256515 31600 raft_consensus.cc:1226] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:36.256969 1763 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:36.236254 1615 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:36.258955 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:36.260106 1615 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:36.262179 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 54760.2us
I20260805 22:24:36.264406 31608 raft_consensus.cc:1226] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
W20260805 22:24:36.284650 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 272925us
I20260805 22:24:36.286376 32529 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.290745 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:36.263980 31486 raft_consensus.cc:2362] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4. Re-sending same reply.
I20260805 22:24:36.297940 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:36.299355 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:36.295725 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:36.301137 32529 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.302304 32529 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.302493 31709 raft_consensus.cc:2379] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:36.303299 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:36.305327 31502 raft_consensus.cc:1226] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:36.306195 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:36.306905 31711 raft_consensus.cc:2379] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:36.308825 1797 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:36.310849 1703 consensus_queue.cc:1046] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:36.311687 1562 raft_consensus.cc:3032] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:36.310220 1797 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:36.312225 1562 raft_consensus.cc:729] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:36.314241 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.321036 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.321764 31502 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:36.323299 31444 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.324342 31444 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.325191 31444 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.303769 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:36.303895 1190 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:36.307677 1379 raft_consensus.cc:2781] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:36.306483 1374 raft_consensus.cc:2726] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:36.332074 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.336172 31502 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:36.299453 1775 consensus_queue.cc:1046] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
I20260805 22:24:36.340404 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.328480 1504 raft_consensus.cc:2726] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:24:36.370482 1768 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:36.371441 1797 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:36.371631 1504 consensus_queue.cc:1046] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:24:36.372684 1768 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:36.373987 1379 raft_consensus.cc:686] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:36.341518 1562 consensus_queue.cc:260] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:36.375186 1788 log.cc:924] Time spent T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.073s user 0.000s sys 0.001s
I20260805 22:24:36.375502 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:36.376307 1478 raft_consensus.cc:3032] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:36.319437 1703 consensus_queue.cc:1046] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.022s
W20260805 22:24:36.376632 1493 consensus_meta.cc:287] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.563s user 0.010s sys 0.000s
I20260805 22:24:36.377027 1493 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:36.378971 1493 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:36.375811 31609 raft_consensus.cc:1226] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:36.329047 1514 raft_consensus.cc:2726] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:36.328862 1540 raft_consensus.cc:2726] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:36.376211 1562 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:36.384009 31502 raft_consensus.cc:2454] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
W20260805 22:24:36.415103 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 110441us
I20260805 22:24:36.418354 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:36.418716 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:36.334632 1699 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:36.420982 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:36.333439 1784 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
W20260805 22:24:36.420900 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 122182us
W20260805 22:24:36.423424 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 96944.3us
I20260805 22:24:36.423121 31556 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.423856 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.376667 1478 raft_consensus.cc:729] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:36.427882 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.434885 31713 raft_consensus.cc:1226] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:36.419636 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.338248 1190 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:36.443774 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.442453 31499 raft_consensus.cc:2454] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:36.452373 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.375048 1379 consensus_queue.cc:237] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:36.453449 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:36.453614 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:36.427737 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.458344 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.430604 1775 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:36.433763 1503 raft_consensus.cc:2726] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:36.461598 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.462278 31501 raft_consensus.cc:1261] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:36.462396 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.462598 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.463902 32585 raft_consensus.cc:3037] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:36.487082 31556 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.497490 31608 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260805 22:24:36.496215 1819 consensus_queue.cc:1046] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260805 22:24:36.499831 1819 raft_consensus.cc:3032] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:36.500177 1819 raft_consensus.cc:729] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:36.500988 1819 consensus_queue.cc:260] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:36.502210 1796 log.cc:924] Time spent T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260805 22:24:36.502305 1819 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:36.503621 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.499680 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:36.504256 31494 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:36.509133 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:36.517048 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.519235 31444 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.460625 31486 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:36.527720 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.529764 1703 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:36.547454 31554 leader_election.cc:398] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Vote denied by peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607) with higher term. Message: Invalid argument: T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3.
I20260805 22:24:36.563225 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.563866 31444 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.565243 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.565677 31554 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.531630 1553 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.036s
I20260805 22:24:36.429083 1784 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:36.511922 1503 raft_consensus.cc:2726] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:36.578836 31554 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.581323 31667 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:36.542376 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.577739 31704 raft_consensus.cc:1226] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:36.584369 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:36.585968 31554 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.585306 31495 raft_consensus.cc:2362] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2. Re-sending same reply.
I20260805 22:24:36.589846 31554 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:36.567013 1540 raft_consensus.cc:2726] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:36.599851 1765 raft_consensus.cc:2781] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:36.600283 1765 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:36.484140 1775 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:36.602272 31554 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:36.570335 1493 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:36.604719 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:36.465253 1478 consensus_queue.cc:260] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:36.607239 31667 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:36.611696 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:36.614907 31667 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:36.600660 1765 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:36.588390 1121 raft_consensus.cc:2726] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:36.592300 1615 raft_consensus.cc:2726] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:36.592064 1634 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:36.624826 1634 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:36.625290 1634 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:36.623243 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:36.629956 32589 raft_consensus.cc:2454] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:36.630872 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.630959 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.632594 31486 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:36.636559 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:36.637832 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.589164 1372 consensus_queue.cc:1035] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.034s
I20260805 22:24:36.643405 31494 raft_consensus.cc:1261] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:36.614200 1493 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:36.609241 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:36.659631 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:36.647635 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:36.648036 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.649700 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.650286 1190 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:36.620096 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:36.650811 1810 log.cc:924] Time spent T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.196s user 0.001s sys 0.000s
I20260805 22:24:36.726243 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:36.651242 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.726928 31704 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:36.653676 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.726552 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:36.656198 1784 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:36.645550 1828 raft_consensus.cc:2726] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:36.614979 1766 consensus_queue.cc:1035] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.065s
I20260805 22:24:36.658290 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.622460 1753 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:36.588780 1771 consensus_queue.cc:1035] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.053s
I20260805 22:24:36.647272 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.673672 31495 raft_consensus.cc:2379] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:36.725153 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.725914 32586 raft_consensus.cc:1226] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:36.737845 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.727208 31492 raft_consensus.cc:2379] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 3: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:36.738273 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.660403 1493 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:36.648113 1782 raft_consensus.cc:2726] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:36.735842 31554 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:36.645938 1478 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:36.725971 1775 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:36.749926 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 135096us
W20260805 22:24:36.751526 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 182883us
I20260805 22:24:36.752022 31667 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:36.752245 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.753469 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.658291 1634 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:36.736755 1753 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:36.757138 31667 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:36.757738 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.761933 31667 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:36.762456 31495 raft_consensus.cc:1261] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:36.763622 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.763827 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:36.764554 31667 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:36.765620 31667 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:36.767306 31494 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:36.763417 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.773249 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:36.742569 31486 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:36.774904 31490 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260805 22:24:36.775188 32583 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260805 22:24:36.777737 1634 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:36.779384 1576 raft_consensus.cc:3032] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:36.779862 1576 raft_consensus.cc:729] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:36.761032 1372 raft_consensus.cc:2726] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:36.769133 1553 raft_consensus.cc:2781] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:36.782692 1553 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:36.783067 1553 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:36.784312 31502 raft_consensus.cc:2396] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260805 22:24:36.785863 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:36.756368 1748 raft_consensus.cc:2781] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:36.788187 31706 raft_consensus.cc:1261] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:24:36.760596 1782 consensus_queue.cc:1035] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.035s
I20260805 22:24:36.794041 1514 raft_consensus.cc:3032] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:36.794659 1514 raft_consensus.cc:729] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:36.796347 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.797640 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.781152 1561 raft_consensus.cc:3032] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:36.798707 1561 raft_consensus.cc:729] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:36.799201 1350 raft_consensus.cc:3032] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:36.799711 1350 raft_consensus.cc:729] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:36.800182 1561 consensus_queue.cc:260] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:36.796131 1514 consensus_queue.cc:260] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:36.800696 1350 consensus_queue.cc:260] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:36.804282 31604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 1973) took 1141 ms. Trace:
I20260805 22:24:36.761754 1753 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:36.809952 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:36.810355 31604 rpcz_store.cc:276] 0805 22:24:35.662433 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:35.662544 (+ 111us) service_pool.cc:227] Handling call
0805 22:24:35.662815 (+ 271us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:36.254989 (+592174us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:36.804029 (+549040us) spinlock_profiling.cc:243] Waited 587 ms on lock 0x7b54000b1b54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:36.804225 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1348965120}
I20260805 22:24:36.803638 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:36.813759 31447 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:36.815479 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 49906.5us
I20260805 22:24:36.818717 31447 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.763464 1775 raft_consensus.cc:2726] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:36.821101 31447 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:36.822510 31667 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:36.824538 1703 raft_consensus.cc:2781] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:36.824460 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:36.826261 1703 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:36.826639 1703 raft_consensus.cc:3037] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:36.822125 1763 raft_consensus.cc:2726] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:36.825163 31447 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.783512 32585 raft_consensus.cc:2454] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:36.800774 1748 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:36.831928 32594 raft_consensus.cc:1226] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:36.834414 31667 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:36.834344 1828 consensus_queue.cc:1046] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:24:36.836285 31447 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.820621 1381 raft_consensus.cc:2726] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:36.837503 31447 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:36.821554 1771 raft_consensus.cc:2726] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:36.837353 31667 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:36.839958 31447 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.842394 31554 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.844923 31447 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:36.836119 1748 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:36.804365 1765 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:36.829983 1561 raft_consensus.cc:3037] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:36.846921 31554 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.826647 31490 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:36.830439 1763 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:36.847960 1140 consensus_queue.cc:1035] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:36.836378 1775 raft_consensus.cc:2781] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:36.835552 1835 raft_consensus.cc:2726] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:36.851876 32586 raft_consensus.cc:1226] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:36.833354 1604 consensus_queue.cc:1035] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.007s
I20260805 22:24:36.847762 1381 raft_consensus.cc:2726] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:36.836004 1514 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:36.853694 31554 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.855044 31667 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:36.823081 1190 consensus_queue.cc:1046] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.019s
I20260805 22:24:36.839453 1540 raft_consensus.cc:2726] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:36.846331 1350 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:36.852398 1803 raft_consensus.cc:2726] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:36.851208 1634 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:36.887238 1634 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:36.887792 1634 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:24:36.889969 1798 log.cc:924] Time spent T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.516s user 0.013s sys 0.000s
I20260805 22:24:36.884814 1784 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:36.873536 1694 raft_consensus.cc:2726] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260805 22:24:36.879663 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 108174us
I20260805 22:24:36.853188 1771 raft_consensus.cc:2726] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:36.861164 1464 raft_consensus.cc:2726] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:36.872663 1775 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:36.878355 1763 raft_consensus.cc:2726] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:36.861521 1372 consensus_queue.cc:1035] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260805 22:24:36.894260 31554 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.881124 31447 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.905262 1733 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:36.908612 31554 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.915023 31554 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.915025 31447 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:36.918545 31554 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.920019 31554 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.892330 1835 raft_consensus.cc:2726] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:36.914067 1775 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:36.928166 31554 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:36.913865 1464 raft_consensus.cc:2726] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260805 22:24:36.929922 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 74970.3us
I20260805 22:24:36.918113 1190 raft_consensus.cc:2726] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:36.917779 1140 raft_consensus.cc:2726] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:36.918882 1784 raft_consensus.cc:2726] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:36.936695 31667 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:36.923378 1763 raft_consensus.cc:2726] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:36.923139 1768 raft_consensus.cc:2726] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:36.930835 1610 consensus_queue.cc:1035] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
W20260805 22:24:36.944952 1836 log.cc:924] Time spent T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20260805 22:24:36.925315 1803 consensus_queue.cc:1046] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260805 22:24:36.947103 1493 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:36.947819 1553 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:36.925175 1733 consensus_queue.cc:1046] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260805 22:24:36.951416 31604 raft_consensus.cc:1226] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:36.928740 1615 raft_consensus.cc:2726] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:36.936654 1604 raft_consensus.cc:2726] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:36.967257 31667 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:36.998926 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:37.045710 31603 raft_consensus.cc:1226] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:37.046119 31495 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
W20260805 22:24:37.046319 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 79163.4us
I20260805 22:24:37.047201 31667 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:37.053867 31499 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:37.055158 1610 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:37.056268 1610 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:36.997964 1748 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.058144 1203 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:37.059423 1748 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:37.059989 1610 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:36.962610 1803 raft_consensus.cc:3032] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Stepping down as leader of term 3
W20260805 22:24:37.063733 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 131699us
W20260805 22:24:36.961548 1793 log.cc:924] Time spent T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.543s user 0.002s sys 0.001s
I20260805 22:24:37.063449 31667 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:37.045432 1763 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:37.060243 1848 raft_consensus.cc:2726] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260805 22:24:37.065207 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 148417us
I20260805 22:24:36.957355 1765 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:36.964206 1493 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:37.059901 1813 log.cc:924] Time spent T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.646s user 0.001s sys 0.000s
I20260805 22:24:37.063163 1350 raft_consensus.cc:2726] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:37.071311 1566 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260805 22:24:36.964421 1634 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:37.069284 1372 raft_consensus.cc:2726] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:37.071872 1763 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.064200 1803 raft_consensus.cc:729] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:37.049721 1478 consensus_queue.cc:1046] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.024s
W20260805 22:24:37.079878 1832 log.cc:924] Time spent T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.321s user 0.001s sys 0.000s
I20260805 22:24:37.059484 1703 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:36.995970 1853 raft_consensus.cc:2726] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:37.090440 32529 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:37.090978 1806 log.cc:924] Time spent T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.678s user 0.003s sys 0.000s
I20260805 22:24:37.095705 1634 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:37.096922 1703 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:37.098296 1553 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:37.099270 1803 consensus_queue.cc:260] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:37.106361 1749 log.cc:924] Time spent T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 1.101s user 0.003s sys 0.000s
I20260805 22:24:37.100121 1733 raft_consensus.cc:2726] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260805 22:24:37.100968 1803 raft_consensus.cc:3037] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:37.113819 1775 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.102135 1478 raft_consensus.cc:3032] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:37.100802 1856 consensus_queue.cc:1035] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260805 22:24:37.114859 1478 raft_consensus.cc:729] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:37.114470 1576 consensus_queue.cc:260] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:37.116364 1576 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:37.116201 1478 consensus_queue.cc:260] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:37.102319 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:37.119051 1478 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:37.108893 1763 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:37.112905 1753 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.119328 1775 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:37.124589 1752 log.cc:924] Time spent T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 1.132s user 0.004s sys 0.000s
W20260805 22:24:37.129182 1791 log.cc:924] Time spent T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.796s user 0.003s sys 0.000s
I20260805 22:24:37.131549 1753 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:37.135097 1290 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:37.144117 1838 raft_consensus.cc:3032] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:37.144522 1838 raft_consensus.cc:729] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:37.145161 1838 consensus_queue.cc:260] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.145908 1782 raft_consensus.cc:3032] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:37.146387 1782 raft_consensus.cc:729] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:37.146672 1838 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:37.147531 1782 consensus_queue.cc:260] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:37.149466 1782 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:37.187915 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.190610 31486 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:37.192610 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:36.951889 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.064626 31554 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:37.196960 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.069684 31667 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:37.205968 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:24:37.207580 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 138239us
I20260805 22:24:37.208762 31667 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:37.209476 31501 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:37.199807 31602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1000) took 1100 ms. Trace:
I20260805 22:24:37.210215 31602 rpcz_store.cc:276] 0805 22:24:36.098914 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:36.181127 (+ 82213us) service_pool.cc:227] Handling call
0805 22:24:36.185930 (+ 4803us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:36.193417 (+ 7487us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:36.193456 (+ 39us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:36.193866 (+ 410us) log.cc:842] Serialized 26 byte log entry
0805 22:24:36.295501 (+101635us) raft_consensus.cc:1631] Marking committed up to 1
0805 22:24:36.295537 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:36.295603 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:36.906287 (+610684us) spinlock_profiling.cc:243] Waited 4.87 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:24:37.198547 (+292260us) raft_consensus.cc:1671] finished
0805 22:24:37.198592 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:37.199534 (+ 942us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7092,"raft.queue_time_us":25337,"raft.run_cpu_time_us":277,"raft.run_wall_time_us":23480,"spinlock_wait_cycles":11193728,"thread_start_us":72581,"threads_started":1,"wal-append.queue_time_us":144775}
I20260805 22:24:37.202481 31601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1001) took 1098 ms. Trace:
W20260805 22:24:37.202630 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 137991us
I20260805 22:24:37.212921 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.203521 31608 raft_consensus.cc:1226] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:37.204006 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:37.205171 31596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 1969) took 1668 ms. Trace:
I20260805 22:24:37.102051 32529 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:37.217911 31596 rpcz_store.cc:276] 0805 22:24:35.536424 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:35.538045 (+ 1621us) service_pool.cc:227] Handling call
0805 22:24:35.538346 (+ 301us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:35.538669 (+ 323us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:35.538696 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:35.574611 (+ 35915us) log.cc:842] Serialized 26 byte log entry
0805 22:24:35.838943 (+264332us) raft_consensus.cc:1631] Marking committed up to 1
0805 22:24:35.838983 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:35.839050 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:37.104584 (+1265534us) spinlock_profiling.cc:243] Waited 56.4 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:24:37.204022 (+ 99438us) raft_consensus.cc:1671] finished
0805 22:24:37.204069 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:37.204870 (+ 801us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":2261,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":134,"spinlock_wait_cycles":129806208,"thread_start_us":182060,"threads_started":1,"wal-append.queue_time_us":334183}
W20260805 22:24:37.223815 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 121697us
I20260805 22:24:37.227166 31500 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:37.212311 31601 rpcz_store.cc:276] 0805 22:24:36.103463 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:36.185534 (+ 82071us) service_pool.cc:227] Handling call
0805 22:24:36.185816 (+ 282us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:36.211784 (+ 25968us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:36.211811 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:36.216618 (+ 4807us) log.cc:842] Serialized 26 byte log entry
0805 22:24:36.278081 (+ 61463us) raft_consensus.cc:1631] Marking committed up to 2
0805 22:24:36.278121 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:36.278282 (+ 161us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:37.128411 (+850129us) spinlock_profiling.cc:243] Waited 1.39 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:24:37.201875 (+ 73464us) raft_consensus.cc:1671] finished
0805 22:24:37.201917 (+ 42us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:37.202285 (+ 368us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31630,"raft.queue_time_us":27295,"raft.run_cpu_time_us":141,"raft.run_wall_time_us":3433,"spinlock_wait_cycles":3206400,"thread_start_us":61104,"threads_started":1,"wal-append.queue_time_us":82221}
I20260805 22:24:37.232915 1753 raft_consensus.cc:2781] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:37.236037 1782 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:37.233593 1771 raft_consensus.cc:2726] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:37.234616 1753 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:37.240228 1753 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:37.236650 1782 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:37.242817 1782 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:37.236631 1807 raft_consensus.cc:2726] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:37.252148 31493 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:37.253643 32529 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:37.254130 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.254941 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.255905 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.256783 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.257970 31490 raft_consensus.cc:2379] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:37.259014 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.259828 31490 raft_consensus.cc:2454] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:37.260890 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.261703 1807 raft_consensus.cc:2726] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:37.266065 31609 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 1975) took 1416 ms. Trace:
I20260805 22:24:37.266552 31609 rpcz_store.cc:276] 0805 22:24:35.849589 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:36.171602 (+322013us) service_pool.cc:227] Handling call
0805 22:24:36.171870 (+ 268us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:36.380353 (+208483us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:37.265517 (+885164us) spinlock_profiling.cc:243] Waited 204 ms on lock 0x7b5400238cd4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:37.265871 (+ 354us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":468620672}
I20260805 22:24:37.269917 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.271381 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.272428 31492 raft_consensus.cc:2379] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:37.279904 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:37.280267 31494 raft_consensus.cc:1226] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:37.280982 31596 raft_consensus.cc:2379] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 3: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:37.279464 1782 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:37.256812 31500 raft_consensus.cc:2362] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2. Re-sending same reply.
I20260805 22:24:37.282796 31493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40740 (request call id 998) took 1106 ms. Trace:
I20260805 22:24:37.283689 31704 raft_consensus.cc:1226] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260805 22:24:37.283216 31493 rpcz_store.cc:276] 0805 22:24:36.175847 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:36.177866 (+ 2019us) service_pool.cc:227] Handling call
0805 22:24:37.282553 (+1104687us) spinlock_profiling.cc:243] Waited 1.07 s on lock 0x7b5400332054. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:37.282715 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2465233792}
I20260805 22:24:37.286510 32529 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:37.298930 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:37.299475 32584 raft_consensus.cc:1226] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:37.302904 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:37.303654 32594 raft_consensus.cc:3037] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:37.305738 1807 raft_consensus.cc:2781] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:37.306315 1807 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:37.306759 1807 raft_consensus.cc:3037] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:37.309741 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:37.310933 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:37.263008 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:37.315635 31705 raft_consensus.cc:2379] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 2: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:37.316309 31705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1578) took 1025 ms. Trace:
I20260805 22:24:37.316658 31705 rpcz_store.cc:276] 0805 22:24:36.291248 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:36.295617 (+ 4369us) service_pool.cc:227] Handling call
0805 22:24:37.310695 (+1015078us) spinlock_profiling.cc:243] Waited 8.73 ms on lock 0x7b58002d0698. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108bce 000000000034aea9 00007f2735f34a30 00007f2735ffa7b4 00007f2735ffa821 00007f2735faa1b0 00007f2735fa9a94 00007f272f691c04 00007f272f691b92
0805 22:24:37.316193 (+ 5498us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20075392}
W20260805 22:24:37.317615 31668 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.457s user 0.006s sys 0.012s
W20260805 22:24:37.318137 31668 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.457s user 0.006s sys 0.012s
W20260805 22:24:37.318513 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 1.45775e+06us
I20260805 22:24:37.310879 1803 consensus_queue.cc:1046] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260805 22:24:37.319870 1803 raft_consensus.cc:3032] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:37.320523 1803 raft_consensus.cc:729] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:37.321915 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:37.322855 31713 raft_consensus.cc:2454] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:37.324785 1771 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:37.327730 1771 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.329713 31490 raft_consensus.cc:2379] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 3: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:37.329867 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:37.322983 1858 consensus_queue.cc:1046] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.008s
I20260805 22:24:37.331712 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:37.332584 31705 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:37.333133 1771 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:37.331912 1858 raft_consensus.cc:3032] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:37.337915 1858 raft_consensus.cc:729] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:37.338111 32586 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260805 22:24:37.338798 31357 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47946 (request call id 183) took 2641 ms. Trace:
I20260805 22:24:37.342350 32579 raft_consensus.cc:1226] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:37.339330 31357 rpcz_store.cc:276] 0805 22:24:34.697296 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:34.697495 (+ 199us) service_pool.cc:227] Handling call
0805 22:24:34.904145 (+206650us) spinlock_profiling.cc:243] Waited 192 ms on lock 0x7b5400002dd4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fedaf0 00007f2731070f2a 00007f2736fc2d6c 00007f27370bcc3e 00007f27353cefb4 00007f27353cef42 00007f27353cee71 00007f27353cedec 00007f27353ce0a2 00007f272f49294c 00007f272f491d86
0805 22:24:37.338546 (+2434401us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0805 22:24:35.778905 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0805 22:24:35.834595 (+ 55690us) write_op.cc:432] Acquiring schema lock in shared mode
0805 22:24:35.836753 (+ 2158us) write_op.cc:435] Acquired schema lock
0805 22:24:35.836772 (+ 19us) tablet.cc:660] Decoding operations
0805 22:24:36.013291 (+176519us) write_op.cc:620] Acquiring the partition lock for write op
0805 22:24:36.028941 (+ 15650us) write_op.cc:641] Partition lock acquired for write op
0805 22:24:36.031596 (+ 2655us) tablet.cc:683] Acquiring locks for 7 operations
0805 22:24:36.168811 (+137215us) tablet.cc:699] Row locks acquired
0805 22:24:36.168827 (+ 16us) write_op.cc:260] PREPARE: finished
0805 22:24:36.168982 (+ 155us) write_op.cc:270] Start()
0805 22:24:36.169095 (+ 113us) write_op.cc:276] Timestamp: P: 1785968676168942 usec, L: 0
0805 22:24:36.169113 (+ 18us) op_driver.cc:331] REPLICATION: starting
0805 22:24:36.169794 (+ 681us) log.cc:842] Serialized 2835 byte log entry
0805 22:24:36.373290 (+203496us) op_driver.cc:444] REPLICATION: finished
0805 22:24:36.819468 (+446178us) spinlock_profiling.cc:243] Waited 7.81 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f272e1225fe 00007f2736f59a39 00007f272e13f8bb 00007f272e13dcbf 00007f272e13fe22 00007f2734a78410
0805 22:24:36.833114 (+ 13646us) write_op.cc:301] APPLY: starting
0805 22:24:36.833195 (+ 81us) tablet.cc:1365] starting BulkCheckPresence
0805 22:24:36.833428 (+ 233us) tablet.cc:1368] finished BulkCheckPresence
0805 22:24:36.833452 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0805 22:24:36.834362 (+ 910us) tablet.cc:1381] finished ApplyRowOperation cycle
0805 22:24:36.834585 (+ 223us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0805 22:24:36.834632 (+ 47us) write_op.cc:312] APPLY: finished
0805 22:24:36.834908 (+ 276us) log.cc:842] Serialized 59 byte log entry
0805 22:24:36.847987 (+ 13079us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:36.848102 (+ 115us) write_op.cc:454] Released schema lock
0805 22:24:36.848160 (+ 58us) write_op.cc:341] FINISH: Updating metrics
0805 22:24:36.849579 (+ 1419us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:36.849612 (+ 33us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":9403,"reported_tablets":11,"spinlock_wait_cycles":441949440,"child_traces":[["op",{"apply.queue_time_us":459544,"apply.run_cpu_time_us":2809,"apply.run_wall_time_us":16827,"num_ops":7,"prepare.queue_time_us":90183,"prepare.run_cpu_time_us":5363,"prepare.run_wall_time_us":440835,"replication_time_us":204076,"spinlock_wait_cycles":17957120,"thread_start_us":556267,"threads_started":2}]]}
I20260805 22:24:37.339283 1858 consensus_queue.cc:260] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:37.359493 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 73358.5us
I20260805 22:24:37.367576 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:37.368036 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:37.368811 1131 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:37.379043 1131 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.380842 1858 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:37.381493 1131 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:37.369364 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:37.322232 1803 consensus_queue.cc:260] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.382941 31495 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:37.383203 1803 raft_consensus.cc:3037] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:37.370477 32579 raft_consensus.cc:1226] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:37.385632 31704 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:37.389962 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.389041 32594 raft_consensus.cc:2454] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:37.391958 31499 raft_consensus.cc:2396] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260805 22:24:37.388847 32586 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:37.396854 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.403507 1754 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:37.404279 1754 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.404608 1574 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260805 22:24:37.408561 1754 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:37.413307 31708 raft_consensus.cc:1226] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:37.423504 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.424410 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:37.425038 31499 raft_consensus.cc:2362] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5. Re-sending same reply.
I20260805 22:24:37.423671 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.426337 31495 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:37.424966 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:37.411808 31356 catalog_manager.cc:5495] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 2 to 3, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:37.435032 31490 raft_consensus.cc:1226] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 2. Current term is 3. Ops: [2.2-2.2]
I20260805 22:24:37.431387 31706 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:37.442700 1478 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:37.443037 1131 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:37.444249 1771 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:37.450690 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:37.440577 31486 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:37.446282 1782 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:37.453945 31708 raft_consensus.cc:1261] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:37.443266 31356 catalog_manager.cc:5495] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 2 to 3, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:37.455834 31492 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260805 22:24:37.459688 31490 raft_consensus.cc:1261] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:24:37.445940 1478 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:37.460781 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:37.461045 1131 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.460791 31554 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:37.461913 31500 raft_consensus.cc:1226] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260805 22:24:37.462050 31490 raft_consensus.cc:2379] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:37.471273 31705 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:37.466773 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:37.471772 1131 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:37.472084 31713 raft_consensus.cc:1226] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:37.452713 1753 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:37.477698 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:37.474004 1754 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:37.478729 1753 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:37.436534 31501 raft_consensus.cc:2932] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:37.482782 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:37.467931 1782 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:37.489482 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:37.490993 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:37.491238 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:37.492795 31556 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:37.488438 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:37.468626 1771 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.500532 1131 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:37.501597 1478 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:37.505364 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:37.508688 1780 raft_consensus.cc:2932] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:37.510810 1771 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:37.499760 1782 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.515738 1807 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.516184 1782 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:37.517762 31447 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:37.520721 31356 catalog_manager.cc:5495] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 1 to 2, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 2 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:37.532001 1478 raft_consensus.cc:2726] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:37.533579 31447 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:37.530572 31356 catalog_manager.cc:5495] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 2 to 3, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:37.536903 1545 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:37.538511 1290 consensus_queue.cc:1046] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:37.543781 1545 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.544557 1803 raft_consensus.cc:3032] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:37.544971 1803 raft_consensus.cc:729] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:37.539494 31447 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:37.540550 1838 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:37.533514 32584 raft_consensus.cc:1226] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:37.545898 1838 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.546805 1545 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:37.529793 31554 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:37.549332 1838 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:37.550242 1753 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:37.553232 1797 raft_consensus.cc:2781] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:37.554668 1838 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:37.554080 31668 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:37.554847 1831 consensus_queue.cc:1046] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:24:37.555462 1838 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.558575 1797 raft_consensus.cc:686] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
W20260805 22:24:37.574353 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 68892.4us
I20260805 22:24:37.576493 1694 raft_consensus.cc:2781] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:37.576985 1694 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:37.577328 1694 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:37.577762 1780 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.ChangeConfig from 127.0.0.1:59024 (request call id 1077) took 2012 ms. Trace:
I20260805 22:24:37.559230 1771 consensus_queue.cc:1046] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.011s
W20260805 22:24:37.588799 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 49385.6us
I20260805 22:24:37.572011 1803 consensus_queue.cc:260] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.590924 1803 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:37.587522 1290 raft_consensus.cc:2781] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:37.597536 1290 raft_consensus.cc:686] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:37.598729 1290 consensus_queue.cc:237] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.600435 1824 consensus_queue.cc:1035] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:37.602671 1545 raft_consensus.cc:3032] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:37.603317 1545 raft_consensus.cc:729] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:37.606124 1699 consensus_queue.cc:1046] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:24:37.607671 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:37.608999 32586 raft_consensus.cc:1226] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:37.610183 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:37.610350 31554 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:37.611120 1754 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:37.612462 31554 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:37.606431 1838 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:37.606520 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:37.613116 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:37.613842 32579 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
W20260805 22:24:37.614198 1863 log.cc:924] Time spent T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.125s user 0.000s sys 0.004s
I20260805 22:24:37.607230 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:37.616670 31356 catalog_manager.cc:5495] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 1 to 2, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 2 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:24:37.610106 1839 raft_consensus.cc:3032] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:37.618000 1839 raft_consensus.cc:729] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:37.617367 32584 raft_consensus.cc:2932] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:37.618976 1839 consensus_queue.cc:260] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.620498 1839 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
W20260805 22:24:37.625507 1864 log.cc:924] Time spent T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.064s user 0.000s sys 0.002s
I20260805 22:24:37.629324 1485 raft_consensus.cc:3032] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:24:37.629824 1485 raft_consensus.cc:729] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:37.630187 1797 consensus_queue.cc:237] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.630761 1485 consensus_queue.cc:260] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.631888 1485 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:37.591537 1807 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:37.629561 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 68598.4us
I20260805 22:24:37.636883 1545 consensus_queue.cc:260] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:37.637555 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 49498.8us
I20260805 22:24:37.648538 1545 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:37.652518 31713 raft_consensus.cc:1261] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:37.639139 1771 consensus_queue.cc:1046] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:24:37.654451 1771 raft_consensus.cc:3032] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:37.654873 1771 raft_consensus.cc:729] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:37.578501 1780 rpcz_store.cc:276] 0805 22:24:35.564820 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:35.565031 (+ 211us) service_pool.cc:227] Handling call
0805 22:24:35.781236 (+216205us) log.cc:842] Serialized 501 byte log entry
0805 22:24:36.230171 (+448935us) spinlock_profiling.cc:243] Waited 8.9 ms on lock 0x7b54003a61d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106e7c5 00007f2731066c1a 00007f2735faa6d9 00007f272f694184 00007f272f694112 00007f272f694041 00007f272f693fbc 00007f272f693272 00007f272f49294c 00007f272f491d86
0805 22:24:37.577491 (+1347320us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":20468992,"thread_start_us":148154,"threads_started":1,"wal-append.queue_time_us":211746,"wal-append.run_cpu_time_us":2815,"wal-append.run_wall_time_us":1209486}
I20260805 22:24:37.644783 1838 consensus_queue.cc:1046] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:24:37.655920 1771 consensus_queue.cc:260] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.656442 1838 raft_consensus.cc:3032] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:37.656811 31339 catalog_manager.cc:5008] ChangeConfig:REMOVE_PEER RPC for tablet ab66683a9e5e46c2b09be4afdee2a08c with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20260805 22:24:37.656831 1838 raft_consensus.cc:729] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:37.658461 1838 consensus_queue.cc:260] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.661993 1754 raft_consensus.cc:2781] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:24:37.662565 1754 raft_consensus.cc:686] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:37.663093 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:37.664240 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 51925us
I20260805 22:24:37.651687 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:37.681137 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:37.663518 1754 consensus_queue.cc:237] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:37.687618 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:37.683270 1838 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:37.693718 31556 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:37.693802 1765 consensus_queue.cc:1046] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:24:37.695123 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.683413 1771 raft_consensus.cc:3037] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:37.704038 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.701705 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:37.705410 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.755861 31554 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:37.772603 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.779273 1478 raft_consensus.cc:2726] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:37.779771 1780 raft_consensus.cc:2781] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:37.780202 1780 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:37.780539 1780 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:37.793169 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:37.793933 32529 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:37.795423 32529 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:37.795765 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:37.796481 1741 raft_consensus.cc:2726] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:37.797127 1741 raft_consensus.cc:2726] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:37.802838 31713 raft_consensus.cc:1226] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:37.793956 31711 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:37.821923 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.822721 31495 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:37.823983 32529 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:37.834059 1694 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:37.836380 1694 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:37.878919 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 240757us
I20260805 22:24:37.897327 31711 raft_consensus.cc:1261] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 2. (term mismatch)
I20260805 22:24:37.897770 31711 pending_rounds.cc:76] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385: Aborting all ops after (but not including) 1
I20260805 22:24:37.900417 31711 pending_rounds.cc:97] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385: Aborting uncommitted NO_OP operation due to leader change: 2.2
I20260805 22:24:37.900748 31711 raft_consensus.cc:2866] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:24:37.902421 31704 raft_consensus.cc:1226] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:24:37.905959 31706 raft_consensus.cc:1226] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:37.908663 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:37.911820 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:37.918473 1856 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:37.919437 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:37.920157 31499 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:37.920027 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:37.920331 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:37.919385 31495 raft_consensus.cc:1226] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:37.925002 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.933497 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:37.918685 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.936944 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:37.938948 31596 raft_consensus.cc:1226] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:37.945830 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:37.920836 31597 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:38.033936 31608 raft_consensus.cc:1226] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:38.052933 31603 raft_consensus.cc:1226] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5. Ops: []
W20260805 22:24:38.060214 1753 consensus_meta.cc:287] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.509s user 0.007s sys 0.010s
I20260805 22:24:38.060686 1753 raft_consensus.cc:2726] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:38.063426 1856 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:38.063901 1856 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:38.078231 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:38.082093 32529 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:38.083752 32529 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:38.057140 31705 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:38.080262 1807 consensus_queue.cc:1035] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.012s
I20260805 22:24:38.075740 1858 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.109732 31499 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
W20260805 22:24:38.107673 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 414013us
I20260805 22:24:38.131152 32529 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:38.097616 1741 raft_consensus.cc:2726] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:38.108450 1478 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.143700 32594 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:38.146759 31668 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:38.147259 31667 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:38.140666 1545 raft_consensus.cc:2781] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260805 22:24:38.159929 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 449485us
I20260805 22:24:38.108925 1290 consensus_queue.cc:1035] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.047s
I20260805 22:24:38.138962 1824 consensus_queue.cc:1046] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260805 22:24:38.158962 1839 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:38.164826 1545 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:38.147044 1858 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.155261 1478 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.179962 1754 raft_consensus.cc:2726] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:38.177626 1545 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:38.184340 1780 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.185109 1574 consensus_queue.cc:1046] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 2.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260805 22:24:38.193965 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 48231.7us
W20260805 22:24:38.223310 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 115607us
I20260805 22:24:38.246418 1780 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:38.250204 1882 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:38.250723 1882 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:38.251091 1882 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:38.246717 31667 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:38.253341 1574 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.246671 1349 raft_consensus.cc:2781] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election won for term 3
W20260805 22:24:38.271472 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 75931.4us
I20260805 22:24:38.263916 1485 raft_consensus.cc:2726] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:38.269973 1349 raft_consensus.cc:686] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:38.263737 1574 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.316771 1290 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:38.316864 1886 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:38.339915 1290 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.340050 1886 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.341533 1803 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:38.342182 1803 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.342463 1886 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:38.316759 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 70102.2us
I20260805 22:24:38.343505 31667 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:38.344137 1803 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:38.320673 1856 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.344638 1803 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:38.344678 31554 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:38.349660 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 206833us
W20260805 22:24:38.353166 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 189376us
I20260805 22:24:38.326821 1858 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:38.328131 1807 raft_consensus.cc:3032] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:38.326205 1478 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:38.325264 1545 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.356105 1886 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:38.355858 1771 raft_consensus.cc:2726] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260805 22:24:38.377995 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 62244.7us
I20260805 22:24:38.373903 1754 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:38.387151 31554 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:38.380298 1886 raft_consensus.cc:686] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:38.370287 1753 raft_consensus.cc:3032] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:38.378559 1858 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.360766 1807 raft_consensus.cc:729] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:38.359982 1803 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.383679 1824 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.405784 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:38.408036 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:38.408869 31609 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:38.391820 1290 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:38.392000 1753 raft_consensus.cc:729] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:38.411250 1797 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:38.394626 1780 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
W20260805 22:24:38.423489 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 65383.8us
I20260805 22:24:38.426853 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:38.417209 1478 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:38.428170 32529 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:38.394968 1754 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.430166 31667 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:38.421519 1856 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:38.423561 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:38.439854 1886 consensus_queue.cc:237] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.440605 31597 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:38.443460 1545 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:38.384238 1349 consensus_queue.cc:237] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:38.433821 1877 log.cc:924] Time spent T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.175s user 0.001s sys 0.001s
I20260805 22:24:38.434412 1909 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.450899 1909 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.452163 31554 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:38.453037 1909 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:38.435738 1131 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:38.459151 1882 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.437588 1574 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:38.439106 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:38.438802 1753 consensus_queue.cc:260] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.445179 31444 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:38.451126 1485 consensus_queue.cc:1046] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:38.435119 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:38.433161 1797 raft_consensus.cc:686] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:38.453267 1771 raft_consensus.cc:2932] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:38.456616 31713 raft_consensus.cc:1226] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:38.455897 32586 raft_consensus.cc:2932] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:38.457504 1891 consensus_queue.cc:1035] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:38.458869 1902 raft_consensus.cc:2726] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:38.459555 1131 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:38.436781 1803 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:38.411556 1824 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.433223 1741 raft_consensus.cc:2781] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:38.474997 31596 raft_consensus.cc:2379] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 3: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
W20260805 22:24:38.435489 1883 log.cc:924] Time spent T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260805 22:24:38.476027 31554 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:38.439915 1754 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:38.461257 1882 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:38.495508 1882 raft_consensus.cc:3032] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:38.495883 1882 raft_consensus.cc:729] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:38.429739 1858 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.496784 1920 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:38.497225 1920 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:38.497605 1920 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:38.476841 1574 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.017s
I20260805 22:24:38.501073 1797 consensus_queue.cc:237] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.489022 31444 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:38.437489 1780 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.490206 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:38.504990 31502 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:38.432984 1478 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:38.493247 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 50366.8us
I20260805 22:24:38.507890 1753 raft_consensus.cc:3037] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:38.492007 1131 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:38.448014 1807 consensus_queue.cc:260] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.523749 1891 raft_consensus.cc:2726] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:38.523904 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:38.509069 1741 raft_consensus.cc:686] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:38.527911 31667 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:38.514076 1909 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:38.519022 1478 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:38.541934 1780 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:38.546413 1824 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:38.547297 1858 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:38.533478 1839 consensus_queue.cc:1159] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "f8e9f4c756da4203917c464e2b179994" responder_term: 2 status { last_received { term: 2 index: 5 } last_committed_idx: 5 last_received_current_leader { term: 2 index: 5 } }
I20260805 22:24:38.549309 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:38.496737 1882 consensus_queue.cc:260] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.539649 1909 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:38.554363 1741 consensus_queue.cc:237] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:38.556442 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 80512.2us
I20260805 22:24:38.559993 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:38.564507 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:38.554860 31502 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:24:38.565312 32579 raft_consensus.cc:1261] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:38.528024 1807 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:38.571132 32587 raft_consensus.cc:2379] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 3: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:38.568051 32584 raft_consensus.cc:2454] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:38.573988 1882 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:38.575400 31447 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:38.576176 1858 raft_consensus.cc:2781] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:38.576601 1858 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:38.576951 1858 raft_consensus.cc:3037] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:38.577919 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:38.568924 1909 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:38.585016 1858 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.588975 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:38.589564 32594 raft_consensus.cc:3037] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:38.591001 1754 consensus_queue.cc:1046] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:38.591954 1765 raft_consensus.cc:3032] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:38.592293 1765 raft_consensus.cc:729] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:38.593065 1765 consensus_queue.cc:260] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.596354 31554 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:38.596843 32594 raft_consensus.cc:2454] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:38.597289 1858 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:38.600497 31603 raft_consensus.cc:1261] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260805 22:24:38.602031 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:38.618392 31447 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:38.601902 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:38.603680 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:38.604784 1765 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:38.610474 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:38.619221 31705 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:38.619122 1858 raft_consensus.cc:2781] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:38.619720 31706 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:38.619863 1858 raft_consensus.cc:686] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:38.620872 1858 consensus_queue.cc:237] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.623805 1909 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.625742 1753 raft_consensus.cc:2726] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:38.627295 31705 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:38.627560 31554 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:38.636649 1753 raft_consensus.cc:2781] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260805 22:24:38.637224 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 127968us
I20260805 22:24:38.642573 31667 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:38.645539 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 89696.4us
I20260805 22:24:38.565362 1920 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.655473 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:38.656060 31596 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:38.642755 1485 raft_consensus.cc:2726] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:38.649065 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:38.659235 31554 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:38.637032 1753 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:38.664294 1906 consensus_queue.cc:1035] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:38.658692 1478 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.666414 1478 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.666891 1925 raft_consensus.cc:2726] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:38.668601 1478 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:38.654590 1754 consensus_queue.cc:1035] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.015s
I20260805 22:24:38.660671 32529 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:38.661651 1891 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.678217 32529 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:38.680773 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:38.682724 1891 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.685045 1891 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:38.685555 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:38.687711 1902 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:38.670444 1131 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.650739 1349 consensus_queue.cc:1046] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260805 22:24:38.679430 1931 raft_consensus.cc:2781] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:38.693987 1931 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:38.694353 1931 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:38.680801 1920 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:38.671130 1753 raft_consensus.cc:3037] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:38.705623 1824 raft_consensus.cc:2781] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:38.671506 1478 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.708338 31596 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:38.706969 32529 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:38.674557 1909 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:38.710989 31444 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:38.718534 31554 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:38.719718 1915 raft_consensus.cc:2781] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:38.693709 1131 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:38.720466 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:38.728443 1754 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.720575 1839 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:38.728976 1754 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.729771 1839 raft_consensus.cc:686] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:38.731031 1754 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:38.731696 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:38.733459 1771 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.733701 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:38.710479 1824 raft_consensus.cc:686] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:38.730670 1839 consensus_queue.cc:237] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.735008 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:38.738546 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:38.710633 1902 raft_consensus.cc:686] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
W20260805 22:24:38.739816 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 77751.5us
I20260805 22:24:38.740525 31713 raft_consensus.cc:2379] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 3: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:38.746905 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:38.747457 32591 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:38.735611 1909 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:24:38.720227 1915 raft_consensus.cc:686] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:38.755896 31556 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:38.758009 31667 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:38.712141 1478 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.759388 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:38.759546 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:38.759003 31667 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:38.761314 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:38.735092 1824 consensus_queue.cc:237] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.760401 1909 raft_consensus.cc:686] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 6 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:38.741351 1771 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.755290 1349 consensus_queue.cc:1046] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20260805 22:24:38.771364 1574 raft_consensus.cc:2781] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:38.783285 31502 raft_consensus.cc:1261] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:38.787217 31492 raft_consensus.cc:1226] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260805 22:24:38.755362 1931 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.788151 1574 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:38.790874 1478 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:38.770233 1780 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:38.792469 1771 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:38.773813 1485 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:38.796844 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:38.791455 31708 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:38.806226 1941 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:38.808697 32586 raft_consensus.cc:1226] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:38.803155 1574 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:38.811489 1915 consensus_queue.cc:237] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.820731 31492 raft_consensus.cc:1226] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4. Ops: [3.3-3.3]
I20260805 22:24:38.822582 1927 raft_consensus.cc:3032] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:24:38.823031 1485 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.824338 1902 consensus_queue.cc:237] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.815110 1131 raft_consensus.cc:2726] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:38.825464 1919 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.840714 1919 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.841929 1780 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.842911 1919 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:38.842804 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:38.843494 31706 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:38.843963 1780 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:38.841585 1931 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:38.831996 1927 raft_consensus.cc:729] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:38.846261 1927 consensus_queue.cc:260] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.847697 1927 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:38.850024 1485 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:38.855446 31706 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:38.841692 31494 raft_consensus.cc:2932] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 4 to 5, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
W20260805 22:24:38.857708 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 98784.8us
I20260805 22:24:38.858434 31667 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:38.856989 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 101149us
I20260805 22:24:38.859311 31667 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:38.859237 32529 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:38.865691 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:38.866334 31492 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:38.867539 31444 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:38.865073 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:38.870632 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:38.871353 1478 raft_consensus.cc:2726] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:38.871672 1920 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.872409 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:38.860114 1131 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:38.838050 1753 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.860392 1485 raft_consensus.cc:2726] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:38.875144 31444 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:38.872418 1920 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.864182 1919 raft_consensus.cc:2781] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:38.873064 31356 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34388 (request call id 278) took 9034 ms. Trace:
I20260805 22:24:38.878944 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:38.880196 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:38.880550 1952 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:38.880357 1919 raft_consensus.cc:686] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:38.880120 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:38.879604 31492 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:38.873183 1886 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.895680 31499 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:38.889777 1950 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:38.888882 31351 catalog_manager.cc:5495] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 2, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 2 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:38.906509 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:38.881230 1131 raft_consensus.cc:686] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:38.877780 31356 rpcz_store.cc:276] 0805 22:24:29.839118 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:29.839235 (+ 117us) service_pool.cc:227] Handling call
0805 22:24:38.872941 (+9033706us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0805 22:24:38.274019 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0805 22:24:38.274653 (+ 634us) write_op.cc:432] Acquiring schema lock in shared mode
0805 22:24:38.274705 (+ 52us) write_op.cc:435] Acquired schema lock
0805 22:24:38.274718 (+ 13us) tablet.cc:660] Decoding operations
0805 22:24:38.275488 (+ 770us) write_op.cc:620] Acquiring the partition lock for write op
0805 22:24:38.275572 (+ 84us) write_op.cc:641] Partition lock acquired for write op
0805 22:24:38.275614 (+ 42us) tablet.cc:683] Acquiring locks for 5 operations
0805 22:24:38.276274 (+ 660us) tablet.cc:699] Row locks acquired
0805 22:24:38.276307 (+ 33us) write_op.cc:260] PREPARE: finished
0805 22:24:38.276469 (+ 162us) write_op.cc:270] Start()
0805 22:24:38.276610 (+ 141us) write_op.cc:276] Timestamp: P: 1785968678276413 usec, L: 0
0805 22:24:38.276644 (+ 34us) op_driver.cc:331] REPLICATION: starting
0805 22:24:38.317584 (+ 40940us) log.cc:842] Serialized 2090 byte log entry
0805 22:24:38.663443 (+345859us) op_driver.cc:444] REPLICATION: finished
0805 22:24:38.781696 (+118253us) write_op.cc:301] APPLY: starting
0805 22:24:38.781829 (+ 133us) tablet.cc:1365] starting BulkCheckPresence
0805 22:24:38.782003 (+ 174us) tablet.cc:1368] finished BulkCheckPresence
0805 22:24:38.782156 (+ 153us) tablet.cc:1370] starting ApplyRowOperation cycle
0805 22:24:38.785152 (+ 2996us) tablet.cc:1381] finished ApplyRowOperation cycle
0805 22:24:38.809553 (+ 24401us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0805 22:24:38.809613 (+ 60us) write_op.cc:312] APPLY: finished
0805 22:24:38.809930 (+ 317us) log.cc:842] Serialized 47 byte log entry
0805 22:24:38.810103 (+ 173us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:38.810219 (+ 116us) write_op.cc:454] Released schema lock
0805 22:24:38.810277 (+ 58us) write_op.cc:341] FINISH: Updating metrics
0805 22:24:38.810595 (+ 318us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:38.810621 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":83,"reported_tablets":17,"child_traces":[["op",{"apply.queue_time_us":109162,"apply.run_cpu_time_us":3711,"apply.run_wall_time_us":29335,"num_ops":5,"prepare.queue_time_us":567683,"prepare.run_cpu_time_us":16474,"prepare.run_wall_time_us":157252,"raft.queue_time_us":165112,"raft.run_cpu_time_us":26966,"raft.run_wall_time_us":77624,"replication_time_us":386696,"thread_start_us":629482,"threads_started":4,"wal-append.queue_time_us":168787}]]}
I20260805 22:24:38.912434 31601 raft_consensus.cc:2379] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 4: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:38.913784 1478 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:38.914498 1858 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.916332 1905 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.926234 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:38.929203 31708 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:38.929872 1949 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.929417 1919 consensus_queue.cc:237] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.863597 1909 consensus_queue.cc:237] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.925081 1858 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.882799 1952 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.926120 1349 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:38.900242 1906 raft_consensus.cc:2726] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:38.908820 31351 catalog_manager.cc:5495] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: config changed from index 3 to 4, term changed from 1 to 4, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to <none>, VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) evicted. New cstate: current_term: 4 committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:38.880934 31486 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:24:38.948648 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:38.920383 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:38.927724 31603 raft_consensus.cc:2932] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 2 to 5, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:38.940949 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:38.942701 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:38.942806 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:38.944693 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:38.947822 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:38.951706 31704 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:38.951076 31601 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:38.950588 31709 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:38.969761 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:38.968660 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:38.971634 32529 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:38.971181 31351 catalog_manager.cc:5495] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 2, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 2 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:38.922977 1886 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.973904 31351 catalog_manager.cc:5495] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 2, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 2 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:38.974967 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:38.980806 31708 raft_consensus.cc:2396] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260805 22:24:38.981249 1955 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.981814 1955 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.982273 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:38.982748 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:38.983377 31711 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:38.983978 1951 consensus_queue.cc:1046] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:38.984913 1951 raft_consensus.cc:3032] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:38.985268 1951 raft_consensus.cc:729] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:38.949625 1349 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.987278 1349 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:38.987847 31713 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:38.970999 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:38.930428 1949 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.982884 31715 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:38.977697 31667 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:38.933717 1478 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:38.986253 1951 consensus_queue.cc:260] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:38.943109 1797 consensus_queue.cc:1046] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.032s
I20260805 22:24:38.994031 31704 raft_consensus.cc:3037] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 5
I20260805 22:24:38.994544 31711 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:38.968204 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:38.999146 31667 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:38.975561 1902 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:38.972615 1839 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:38.933522 1905 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.006618 1920 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:39.007331 1905 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:38.919045 1950 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.009495 1950 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:38.987859 1858 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:38.997843 1478 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:38.978878 1485 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:39.003648 1803 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:39.004789 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:39.015837 1858 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:39.016016 1803 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.016364 1858 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.005592 1839 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:39.017488 1839 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:39.018252 1803 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:39.018446 1858 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:39.005754 1902 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.019936 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.020561 31608 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:39.021869 1952 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:39.000591 1753 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:39.000407 1131 consensus_queue.cc:237] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.023819 1824 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:39.024377 31608 raft_consensus.cc:1261] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260805 22:24:39.015487 1485 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.026533 31608 raft_consensus.cc:1226] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:39.027662 1485 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:39.028751 31608 raft_consensus.cc:1226] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:39.013180 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.010156 1797 consensus_queue.cc:1225] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer 6bf8c694a0ba40978195ae044a917caf log is divergent from this leader: its last log entry 5.8 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260805 22:24:38.983933 1955 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:38.998998 1927 raft_consensus.cc:2726] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:39.021943 1694 raft_consensus.cc:3032] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:39.037729 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.038370 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.045110 1886 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:39.046658 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.047063 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.020468 1902 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:39.057320 31707 raft_consensus.cc:2396] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for term 4 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260805 22:24:39.061126 31444 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:39.039683 1950 raft_consensus.cc:2781] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:39.064396 31667 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:39.039940 1951 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:24:39.062951 1959 log.cc:924] Time spent T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.082s user 0.000s sys 0.003s
I20260805 22:24:39.024335 1824 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:39.050467 1694 raft_consensus.cc:729] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:39.122334 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.085640 31444 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:39.061430 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.063393 1920 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:39.063304 1803 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:39.041812 1485 consensus_queue.cc:1046] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260805 22:24:39.151346 31704 raft_consensus.cc:2454] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:39.151403 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.071424 1906 raft_consensus.cc:2781] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:39.125427 31444 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:39.154445 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:39.125087 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.078642 1955 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:39.148975 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.074621 1902 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:39.075438 1950 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:39.061698 1882 consensus_queue.cc:1046] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.076s
I20260805 22:24:39.161346 1950 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:39.130463 1858 raft_consensus.cc:2781] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:39.082893 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.081718 1349 raft_consensus.cc:2781] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:39.129848 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:39.080385 1741 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:39.153580 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.154701 1906 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:39.165680 31603 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:39.176467 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:39.183487 31494 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:39.191509 31501 raft_consensus.cc:3037] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:39.155398 31597 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:39.156595 1941 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:39.169965 32586 raft_consensus.cc:1261] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260805 22:24:39.170109 1349 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:39.164748 1858 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:39.199553 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.206343 31602 raft_consensus.cc:2379] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 3: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:39.153568 1803 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:39.204169 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:39.152866 1920 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.124356 32579 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:60186
I20260805 22:24:39.200449 1915 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:39.211791 32591 raft_consensus.cc:1261] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260805 22:24:39.200035 1906 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:39.215723 1966 raft_consensus.cc:2781] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:39.210636 1858 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:39.219363 31715 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:39.190883 1741 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:39.221436 1803 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:39.222872 1824 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:39.205536 1349 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:39.175634 1574 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:39.198719 1905 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:39.224480 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.224979 31501 raft_consensus.cc:1261] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:24:39.228964 31704 raft_consensus.cc:1261] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:24:39.176064 1955 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.193646 1902 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } attrs { replace: true } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:39.227106 1966 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:39.233098 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.219805 1949 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:39.216316 1920 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:39.226379 1915 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.238166 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.262629 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:39.238897 1839 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:39.226598 32579 raft_consensus.cc:595] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:39.267982 32579 raft_consensus.cc:603] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:24:39.268294 1920 consensus_queue.cc:1035] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
W20260805 22:24:39.244475 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 119802us
I20260805 22:24:39.270169 31444 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:39.271950 1839 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:39.272377 31444 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:39.255097 1951 consensus_queue.cc:1035] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.018s
I20260805 22:24:39.256852 31709 raft_consensus.cc:2362] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5. Re-sending same reply.
I20260805 22:24:39.263221 31602 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 3
I20260805 22:24:39.205490 1941 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.264628 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.264827 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:39.264632 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:39.289597 31714 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:39.291656 31494 raft_consensus.cc:2454] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:39.294149 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:39.260428 1905 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.225262 1478 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:39.245692 1741 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:39.269661 1958 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:39.273543 1920 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:39.303047 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:39.263909 1966 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:39.265342 1824 consensus_queue.cc:1035] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.026s
I20260805 22:24:39.282547 1574 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:39.252066 1803 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:39.289964 31554 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:39.273474 1955 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:39.299175 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.274027 31444 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:39.278545 31667 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:39.306049 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.283054 1780 raft_consensus.cc:3032] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:39.308068 31554 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:39.301297 1919 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:39.321523 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:39.322114 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.322223 31500 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:39.320118 31554 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:39.320261 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 45275.2us
I20260805 22:24:39.323717 32529 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:39.322638 31601 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:24:39.301709 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:39.319348 31499 raft_consensus.cc:2379] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 4: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:39.320160 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:39.325837 31554 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:39.314800 1807 consensus_queue.cc:1035] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.009s
W20260805 22:24:39.324282 1964 log.cc:924] Time spent T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.313s user 0.000s sys 0.003s
I20260805 22:24:39.326659 1949 raft_consensus.cc:2726] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:39.327221 1941 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:39.327162 31667 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:39.327965 1856 raft_consensus.cc:2781] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:39.326609 31444 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:39.358773 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:39.358958 31667 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:39.349181 1888 raft_consensus.cc:2726] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:39.360325 1485 raft_consensus.cc:2726] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:39.337463 32529 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:39.346014 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:39.300962 1478 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:39.348577 1927 raft_consensus.cc:3032] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:39.348752 1915 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:39.349262 32584 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:60186
I20260805 22:24:39.365363 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:39.350584 1902 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:39.346792 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:39.308842 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:39.318271 1803 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:39.382336 31490 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:39.383247 1950 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:39.349041 1905 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:39.390239 31714 raft_consensus.cc:1226] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:39.390661 31490 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
I20260805 22:24:39.391676 31499 raft_consensus.cc:2454] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:39.391357 1950 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:39.344532 31554 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:39.360368 31444 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:39.362389 31500 raft_consensus.cc:1261] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260805 22:24:39.396680 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.397614 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.398177 31444 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:39.304610 1909 raft_consensus.cc:2726] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:24:39.318917 1780 raft_consensus.cc:729] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:39.407363 1780 consensus_queue.cc:260] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:39.325016 1574 raft_consensus.cc:2726] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:39.364032 1831 raft_consensus.cc:2726] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:39.371280 1980 raft_consensus.cc:2726] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:39.372723 31494 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260805 22:24:39.375415 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.375839 32584 raft_consensus.cc:595] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:39.441237 31709 raft_consensus.cc:2354] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3.
I20260805 22:24:39.375433 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.377810 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.442030 31713 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:39.378261 31705 raft_consensus.cc:1226] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 5. Current term is 6. Ops: []
I20260805 22:24:39.379285 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.378523 31486 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:39.377480 1478 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:39.383698 1803 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:39.377508 1906 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.384473 1981 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:39.365430 1927 raft_consensus.cc:729] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:39.318015 1754 raft_consensus.cc:2726] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:39.306702 1958 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.364171 1920 raft_consensus.cc:2726] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:39.362416 1856 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:39.378607 1694 consensus_queue.cc:260] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.343976 1919 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:39.399847 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 55413us
I20260805 22:24:39.402446 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.408293 1988 raft_consensus.cc:2781] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:39.364298 1951 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:39.414434 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.381822 31667 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:39.352995 32586 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:39.376394 31708 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:39.401329 1905 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:39.405917 1753 raft_consensus.cc:2781] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:39.465553 31706 raft_consensus.cc:1226] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:39.466375 31602 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:39.408488 1780 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:39.480691 1753 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:39.471587 31715 raft_consensus.cc:2454] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:39.482304 1919 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:39.482894 1856 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:39.482764 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:39.483629 31597 raft_consensus.cc:2379] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 3: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:39.481756 31601 raft_consensus.cc:2379] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 3: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:39.469565 1982 raft_consensus.cc:2726] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:39.443166 32584 raft_consensus.cc:603] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:24:39.408852 1839 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:39.482017 1905 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.469400 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:39.489495 31493 raft_consensus.cc:2379] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:39.469120 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:39.443431 1950 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:39.481154 1753 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:39.465605 1771 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
W20260805 22:24:39.485713 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 104044us
W20260805 22:24:39.492391 1984 log.cc:924] Time spent T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20260805 22:24:39.492728 1839 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:39.493736 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.487926 1941 raft_consensus.cc:2781] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:39.494750 1941 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:39.494843 1839 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:39.495122 1941 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:39.475025 1981 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:39.496142 1981 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:39.493331 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.485239 1988 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:39.497967 31709 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:39.498152 1988 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:39.489524 31554 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:39.499998 1915 raft_consensus.cc:2726] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:39.484686 1951 raft_consensus.cc:686] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:39.381644 1349 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.494393 31713 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:39.501952 31554 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:39.494335 32587 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:39.503419 31603 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:39.506744 31554 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:24:39.511102 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 148539us
I20260805 22:24:39.504292 1754 raft_consensus.cc:2726] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:39.513099 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:39.515702 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 116539us
I20260805 22:24:39.518699 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.518966 31447 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:39.519394 31708 raft_consensus.cc:1261] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:24:39.523882 31709 raft_consensus.cc:2379] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:39.524322 31713 raft_consensus.cc:1261] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260805 22:24:39.523710 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.528290 31444 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:39.528800 31554 leader_election.cc:398] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3.
I20260805 22:24:39.529079 31554 leader_election.cc:401] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Cancelling election due to peer responding with higher term
I20260805 22:24:39.511780 1694 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:39.534739 31554 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:39.515362 1797 raft_consensus.cc:2781] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:39.541146 31603 raft_consensus.cc:1226] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:39.543269 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.546707 31499 raft_consensus.cc:1261] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:39.479813 1927 consensus_queue.cc:260] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.552660 31667 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:39.547940 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:39.549265 31444 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:39.528288 1982 raft_consensus.cc:2781] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:39.590852 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:39.591374 1982 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:39.591598 31494 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:39.536329 1951 consensus_queue.cc:237] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.592695 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.593799 31495 raft_consensus.cc:1226] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:39.595016 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:39.509181 1771 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:39.535899 1803 raft_consensus.cc:2781] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:39.602747 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.551350 1985 consensus_queue.cc:1035] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260805 22:24:39.590447 1831 consensus_queue.cc:1046] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:24:39.596551 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.506932 1950 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:39.599382 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 50153.9us
I20260805 22:24:39.583531 31554 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:39.562274 31667 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:39.544767 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.535660 1754 raft_consensus.cc:2726] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 2. Reason: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3.
I20260805 22:24:39.602231 31447 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:39.603552 31713 raft_consensus.cc:2379] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 3: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:39.547814 1949 raft_consensus.cc:2781] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260805 22:24:39.627408 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 65223.6us
I20260805 22:24:39.633620 31667 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:39.591853 1982 raft_consensus.cc:3037] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:39.608240 1803 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:39.542176 1902 consensus_queue.cc:1046] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.077s
I20260805 22:24:39.640574 31711 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:39.641095 31554 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:39.551838 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.627918 1574 raft_consensus.cc:2726] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:39.597484 1882 raft_consensus.cc:2726] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:39.624454 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:39.545068 1797 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:39.642230 1996 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:24:39.681638 32587 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260805 22:24:39.601990 1765 consensus_queue.cc:1035] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.036s
I20260805 22:24:39.635612 1980 raft_consensus.cc:2726] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:39.596112 1886 consensus_queue.cc:1046] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.044s
I20260805 22:24:39.639986 1485 raft_consensus.cc:2781] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260805 22:24:39.727054 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 124912us
I20260805 22:24:39.637704 1949 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:39.682318 31705 raft_consensus.cc:2379] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 4: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:39.608426 1931 consensus_queue.cc:1046] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.069s
I20260805 22:24:39.661818 1803 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:24:39.751652 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 131940us
I20260805 22:24:39.771389 31444 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:39.761996 1949 raft_consensus.cc:3037] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:39.779124 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:39.774168 1996 raft_consensus.cc:686] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 6 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:39.662111 1925 consensus_queue.cc:1046] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.099s
I20260805 22:24:39.723242 31715 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 1.
I20260805 22:24:39.721091 32586 raft_consensus.cc:1203] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Deduplicated request from leader. Original: 3.6->[4.7-4.7] Dedup: 4.7->[]
I20260805 22:24:39.727733 1915 raft_consensus.cc:2726] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260805 22:24:39.762367 1858 consensus_meta.cc:287] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.510s user 0.009s sys 0.001s
I20260805 22:24:39.791980 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:39.792918 1994 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:39.728828 1797 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:39.730558 1485 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260805 22:24:39.800422 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 165458us
I20260805 22:24:39.794198 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 2
I20260805 22:24:39.758219 1906 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:39.802963 31708 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:39.803500 31714 raft_consensus.cc:1261] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260805 22:24:39.769837 1131 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:39.733928 1913 consensus_queue.cc:1035] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.062s
I20260805 22:24:39.796449 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:39.797587 1905 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:39.822674 31706 raft_consensus.cc:1261] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:24:39.824200 31601 raft_consensus.cc:1261] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 3. (term mismatch)
I20260805 22:24:39.816604 31668 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:39.824533 31601 pending_rounds.cc:76] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c: Aborting all ops after (but not including) 2
I20260805 22:24:39.824885 31601 pending_rounds.cc:97] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c: Aborting uncommitted NO_OP operation due to leader change: 3.3
I20260805 22:24:39.799918 1958 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:39.807106 1349 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:39.825156 31601 raft_consensus.cc:2866] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:24:39.808676 1485 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:39.805126 1994 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:39.809720 1951 raft_consensus.cc:3032] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:39.810734 32586 raft_consensus.cc:1226] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5. Ops: [4.7-4.7]
I20260805 22:24:39.843613 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:24:39.834419 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:39.853529 31500 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:39.854489 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:39.827108 1478 raft_consensus.cc:2781] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:39.858351 31667 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:39.862071 31667 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:39.857327 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 132342us
I20260805 22:24:39.830567 1925 raft_consensus.cc:2726] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:39.809122 1886 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.036s
I20260805 22:24:39.804912 1858 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.865648 1986 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:39.843948 1994 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:39.844635 1951 raft_consensus.cc:729] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:39.842114 1950 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
W20260805 22:24:39.879776 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 79035.6us
I20260805 22:24:39.819355 1131 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:39.887603 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:39.883344 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 112514us
I20260805 22:24:39.869479 1975 raft_consensus.cc:2781] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:39.888232 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:39.888947 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:39.893404 31601 raft_consensus.cc:2379] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 3: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:39.869525 1882 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:39.868396 1478 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:39.890229 31493 raft_consensus.cc:2454] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:39.886977 31603 raft_consensus.cc:1226] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:39.891686 31444 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:39.902387 31444 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:39.903786 31444 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:39.831939 1949 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:39.905869 1949 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:39.879073 1771 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:39.873499 31705 raft_consensus.cc:1261] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260805 22:24:39.915892 31444 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:39.898423 32529 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:39.868917 1955 consensus_queue.cc:1035] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.031s
I20260805 22:24:39.909543 2004 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:39.923075 32529 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:39.898593 1975 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:39.907300 1478 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:39.946599 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:39.948279 31602 raft_consensus.cc:1261] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:24:39.919730 1909 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:39.975761 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:24:39.982512 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 91616.2us
I20260805 22:24:39.896703 1906 consensus_queue.cc:1035] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.036s
I20260805 22:24:39.880357 1986 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:39.940778 1982 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:39.927933 1931 raft_consensus.cc:2781] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:39.931336 1919 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:24:39.932785 1975 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:39.931078 1920 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:39.896912 1996 consensus_queue.cc:237] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.001904 1931 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260805 22:24:40.002877 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 138085us
I20260805 22:24:40.003090 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:40.005208 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:39.988569 31667 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:39.976086 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:40.032758 1999 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:39.979723 1905 consensus_queue.cc:1035] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.037s
W20260805 22:24:40.036203 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 52932.9us
I20260805 22:24:40.026628 2005 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:39.995676 1955 consensus_queue.cc:1035] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.030s
I20260805 22:24:39.931170 2004 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.026670 1919 raft_consensus.cc:686] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:40.032186 31707 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
W20260805 22:24:40.034240 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 45769.7us
W20260805 22:24:40.046178 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 114054us
W20260805 22:24:40.045001 1856 consensus_meta.cc:287] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.557s user 0.007s sys 0.003s
I20260805 22:24:40.032303 1931 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:40.053895 32529 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:40.050792 31444 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:40.050966 31492 raft_consensus.cc:1261] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:24:40.051694 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:40.055531 2011 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:40.055677 2015 consensus_queue.cc:1046] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:24:40.056051 2011 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.052022 31554 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:39.980602 1951 consensus_queue.cc:260] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.073688 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:40.064724 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:40.057248 1981 consensus_meta.cc:287] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.561s user 0.008s sys 0.000s
I20260805 22:24:40.078416 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:40.026860 1991 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.107s
I20260805 22:24:40.075088 31708 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:24:40.081844 31492 raft_consensus.cc:1261] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:24:40.090358 2018 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:40.090838 2018 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:40.091166 2018 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
W20260805 22:24:40.093653 1997 log.cc:924] Time spent T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.198s user 0.001s sys 0.000s
I20260805 22:24:40.093830 2012 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:40.094182 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:40.094822 2011 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:40.093657 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:40.096711 31714 raft_consensus.cc:3037] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:40.100919 31500 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:40.101577 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:40.101670 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:40.110674 1941 consensus_meta.cc:287] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.615s user 0.010s sys 0.000s
I20260805 22:24:40.114931 1941 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.115909 1982 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:40.082636 1858 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:40.093493 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:40.112231 1831 raft_consensus.cc:3032] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:40.113013 2022 raft_consensus.cc:2781] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:40.119400 31714 raft_consensus.cc:2454] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:40.113574 1919 consensus_queue.cc:237] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.114434 2020 raft_consensus.cc:2781] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election won for term 2
W20260805 22:24:40.111894 2019 log.cc:924] Time spent T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
W20260805 22:24:40.101931 2003 log.cc:924] Time spent T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.176s user 0.000s sys 0.001s
I20260805 22:24:40.096562 1927 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:40.117755 31709 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:40.121002 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:40.140120 31714 raft_consensus.cc:2379] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 3: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:40.120522 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:40.128159 31444 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:40.072803 1925 consensus_queue.cc:1159] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "6bf8c694a0ba40978195ae044a917caf" responder_term: 3 status { last_received { term: 3 index: 6 } last_committed_idx: 5 last_received_current_leader { term: 3 index: 6 } }
W20260805 22:24:40.118517 1694 consensus_meta.cc:287] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.567s user 0.006s sys 0.001s
W20260805 22:24:40.120774 1966 consensus_meta.cc:287] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.805s user 0.006s sys 0.006s
I20260805 22:24:40.090370 31715 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260805 22:24:40.053407 1856 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.165272 31704 raft_consensus.cc:1261] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260805 22:24:40.116889 1941 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:40.119586 2022 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:40.119653 1831 raft_consensus.cc:729] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
W20260805 22:24:40.167972 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 114144us
I20260805 22:24:40.049141 1998 consensus_queue.cc:1035] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.101s
I20260805 22:24:40.078367 1765 consensus_queue.cc:1035] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.070s
W20260805 22:24:40.163553 1988 consensus_meta.cc:287] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.661s user 0.006s sys 0.004s
I20260805 22:24:40.085583 1955 consensus_queue.cc:1035] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.077s
I20260805 22:24:40.184505 32591 raft_consensus.cc:2354] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:24:40.171589 31709 raft_consensus.cc:1226] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 5. Current term is 6. Ops: [5.4-5.4]
I20260805 22:24:40.151707 2020 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:40.147984 1858 raft_consensus.cc:2726] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:40.177990 2015 consensus_queue.cc:1035] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:24:40.084797 1981 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.184000 1988 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.188943 1856 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:40.189690 1989 raft_consensus.cc:3032] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:40.190614 1909 raft_consensus.cc:2726] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:40.203122 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:40.171466 2022 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:40.207746 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:40.210814 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:40.211333 31715 raft_consensus.cc:1226] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:40.216389 31709 raft_consensus.cc:1226] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:40.215359 31500 raft_consensus.cc:3037] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:40.219496 2004 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:40.221504 1797 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.222266 31490 raft_consensus.cc:1226] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:40.222389 2022 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.217043 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:40.224116 31603 raft_consensus.cc:2379] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:40.166124 1966 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.199676 2020 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:24:40.233166 31500 raft_consensus.cc:2454] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:40.233008 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:40.223600 1797 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:40.208231 1989 raft_consensus.cc:729] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:40.182488 1545 consensus_queue.cc:1035] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.043s
I20260805 22:24:40.235699 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:40.236706 1981 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:40.237232 1920 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:40.238180 1574 raft_consensus.cc:3032] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:40.242686 31601 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:40.225486 1986 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:40.252265 31501 raft_consensus.cc:1226] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:40.255364 31667 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:40.257058 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:40.258620 31709 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 3
I20260805 22:24:40.259608 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:40.232348 31715 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:40.261972 31601 raft_consensus.cc:1261] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260805 22:24:40.235715 1988 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:40.254470 32585 raft_consensus.cc:1226] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260805 22:24:40.262389 31486 raft_consensus.cc:2379] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 3: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:40.257481 1574 raft_consensus.cc:729] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:40.275245 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:40.273878 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:40.275924 31714 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:40.282580 31495 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 4
I20260805 22:24:40.291760 32529 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:40.268599 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:40.292701 31554 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:40.296005 31554 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:40.260767 1920 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.275794 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:40.302762 31554 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:40.223634 1831 consensus_queue.cc:260] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.304087 31554 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:40.270486 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:40.295328 2022 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:40.301878 31600 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:40.313558 31604 raft_consensus.cc:1261] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20260805 22:24:40.313925 2021 raft_consensus.cc:2781] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:40.314473 2021 raft_consensus.cc:686] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:40.308028 1574 consensus_queue.cc:260] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.298839 2031 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:40.315389 2021 consensus_queue.cc:237] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.316396 1902 raft_consensus.cc:2781] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:40.316956 2032 raft_consensus.cc:2781] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:40.318626 1949 consensus_queue.cc:1046] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:40.302245 1919 raft_consensus.cc:2781] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:40.327275 1765 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:24:40.327833 1919 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:40.328239 1919 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:40.329255 1839 raft_consensus.cc:3032] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:40.329656 1839 raft_consensus.cc:729] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:40.321655 1902 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:40.330196 1902 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:40.330462 1839 consensus_queue.cc:260] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.331672 1839 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:40.331972 1989 consensus_queue.cc:260] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.314661 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:40.333675 2032 raft_consensus.cc:686] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:40.334168 2012 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:40.334410 1888 raft_consensus.cc:3032] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:24:40.334616 2012 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:40.337666 1966 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:40.320858 1753 consensus_meta.cc:287] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.807s user 0.007s sys 0.004s
I20260805 22:24:40.340337 1753 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.342455 31596 raft_consensus.cc:2379] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 3: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:40.333328 1989 raft_consensus.cc:3037] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:40.313611 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:40.305008 1797 raft_consensus.cc:2781] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:40.343536 1797 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:40.343883 1797 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:40.345870 2005 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:40.319739 1478 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.346446 2005 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.347935 31492 raft_consensus.cc:1226] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:40.349709 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:40.350319 31492 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:40.351678 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:40.302845 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:40.370079 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 48245.9us
I20260805 22:24:40.370728 31601 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260805 22:24:40.371776 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:40.372237 31709 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:40.375959 32529 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:40.376426 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:40.377193 31709 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:40.380321 31495 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:40.324726 1574 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:40.326058 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:40.356657 31667 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:40.334847 1888 raft_consensus.cc:729] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:40.359280 2030 consensus_queue.cc:1046] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:24:40.362483 1931 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.389338 2030 raft_consensus.cc:3032] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:40.389814 2030 raft_consensus.cc:729] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:40.363299 1920 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:40.334674 2032 consensus_queue.cc:237] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.367612 2020 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.385023 31444 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:40.386994 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:40.350205 1131 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:40.374672 1915 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:40.393214 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:40.398451 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:40.392015 31447 leader_election.cc:398] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Vote denied by peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801) with higher term. Message: Invalid argument: T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:24:40.399326 31447 leader_election.cc:401] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260805 22:24:40.397274 31596 raft_consensus.cc:2396] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for term 6 because replica has last-logged OpId of term: 4 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260805 22:24:40.399008 31707 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:40.400309 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:40.389250 1753 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:40.382328 1982 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:40.390741 1741 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:40.390286 2022 consensus_queue.cc:1035] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260805 22:24:40.390659 2023 consensus_queue.cc:1035] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260805 22:24:40.404862 1888 consensus_queue.cc:260] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.407150 32584 raft_consensus.cc:2354] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 2. Current term is 3.
I20260805 22:24:40.385188 31490 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:40.391000 1927 raft_consensus.cc:2781] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:40.408892 31447 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:40.401942 1909 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:40.410351 31709 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:40.405921 1915 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.528031 1902 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.528586 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:40.529516 1915 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:40.529779 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:40.530349 31709 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:40.414178 1931 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:40.414703 2020 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:40.415071 1920 raft_consensus.cc:2726] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801) with higher term. Message: Invalid argument: T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:24:40.418579 2025 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.011s
I20260805 22:24:40.418956 31715 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:40.419533 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:40.382143 1913 consensus_queue.cc:1046] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.020s
I20260805 22:24:40.516672 1131 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:40.530078 1902 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:40.396155 1966 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:40.479530 2007 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:40.482139 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:40.396528 31600 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
I20260805 22:24:40.390888 1886 consensus_queue.cc:1035] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20260805 22:24:40.405871 1858 raft_consensus.cc:2781] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:40.483976 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:40.485100 31502 raft_consensus.cc:1226] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:40.574944 32529 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:40.405090 2005 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:40.410157 31601 raft_consensus.cc:1226] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
W20260805 22:24:40.520217 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 111401us
I20260805 22:24:40.405041 2030 consensus_queue.cc:260] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.411795 1478 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:40.412258 1741 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:40.532836 31444 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:40.474542 1765 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:40.597523 1765 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.548408 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:40.599877 1831 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:40.601781 2010 consensus_queue.cc:1046] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:40.419330 2014 consensus_queue.cc:1046] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.139s
W20260805 22:24:40.592581 1485 consensus_meta.cc:287] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.722s user 0.003s sys 0.008s
I20260805 22:24:40.415006 1975 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.557744 1131 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.533232 2011 raft_consensus.cc:2726] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:40.527508 1927 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:40.407828 1958 consensus_queue.cc:1046] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.125s
I20260805 22:24:40.622035 1958 raft_consensus.cc:3032] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:40.622399 1958 raft_consensus.cc:729] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:40.565745 31709 raft_consensus.cc:2454] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:40.538623 1988 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:40.569793 31596 raft_consensus.cc:1226] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:40.570508 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:40.486030 1982 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.617434 1858 raft_consensus.cc:686] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:40.625087 1988 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.625298 1958 consensus_queue.cc:260] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.625602 1858 consensus_queue.cc:237] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.626607 1958 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:40.627080 1988 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:40.560246 2018 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.699214 1780 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:40.699815 1780 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.700839 2018 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:40.591287 31501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1577) took 1267 ms. Trace:
I20260805 22:24:40.629710 1478 raft_consensus.cc:2781] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:40.599938 1765 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:40.701745 31501 rpcz_store.cc:276] 0805 22:24:39.309630 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:39.311196 (+ 1566us) service_pool.cc:227] Handling call
0805 22:24:39.331499 (+ 20303us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:40.257994 (+926495us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:40.575295 (+317301us) spinlock_profiling.cc:243] Waited 920 ms on lock 0x7b5400121454. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:40.576640 (+ 1345us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2117071744}
I20260805 22:24:40.703125 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:40.704866 1975 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:40.706082 31608 raft_consensus.cc:1226] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5. Ops: []
W20260805 22:24:40.563807 1994 consensus_meta.cc:287] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.670s user 0.003s sys 0.008s
I20260805 22:24:40.706576 1982 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:40.578397 2007 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.594372 31667 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:40.606773 31502 raft_consensus.cc:1226] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 6 LEADER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 5. Current term is 6. Ops: []
I20260805 22:24:40.481600 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:40.541141 1909 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:40.737771 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 118887us
I20260805 22:24:40.597160 1741 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:40.604286 32591 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
W20260805 22:24:40.617818 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 85074.7us
I20260805 22:24:40.557134 1919 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.589684 1915 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:40.616081 2023 raft_consensus.cc:2781] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:40.550813 1888 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:40.617390 1485 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.623437 2030 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:40.737978 31711 raft_consensus.cc:1226] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:40.739353 31707 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:40.615979 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:40.741894 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:40.758200 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:40.603138 2004 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:40.740154 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:40.759505 31502 raft_consensus.cc:2379] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:40.760417 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:40.761150 31502 raft_consensus.cc:2379] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 3: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:40.610941 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:40.609730 2005 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:40.751259 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:40.609935 2026 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:40.623800 1927 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:40.713908 31501 raft_consensus.cc:2379] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 3: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:40.707449 1951 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:40.761945 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:40.578033 1966 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.776558 31706 raft_consensus.cc:2379] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:40.767983 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:40.769469 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:24:40.777829 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 183524us
I20260805 22:24:40.778671 31667 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:40.706887 1994 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.766364 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:40.745404 1803 consensus_meta.cc:287] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.954s user 0.010s sys 0.009s
I20260805 22:24:40.621258 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:40.797503 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:40.792989 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 53713.6us
I20260805 22:24:40.757472 1915 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.794559 2020 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:40.801208 1915 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:40.802227 1131 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:40.802798 32579 raft_consensus.cc:2354] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:24:40.773137 1478 raft_consensus.cc:686] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:40.804523 1478 consensus_queue.cc:237] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.796484 2026 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.810662 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:40.759425 2018 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
W20260805 22:24:40.824231 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 62060.4us
I20260805 22:24:40.825713 31554 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:40.772298 31715 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 4
W20260805 22:24:40.887042 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 87232.6us
I20260805 22:24:40.809158 2026 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:40.946133 32529 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:40.947259 32529 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:40.904893 1478 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
W20260805 22:24:40.911540 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 139907us
I20260805 22:24:40.772434 2011 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:40.923803 31494 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:24:40.793000 1780 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:40.930644 31502 raft_consensus.cc:2454] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:40.798537 1975 raft_consensus.cc:2726] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:40.956875 31447 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:40.961074 31447 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:40.961987 2061 consensus_queue.cc:1046] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:40.813957 2023 raft_consensus.cc:686] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:40.971777 2033 raft_consensus.cc:2781] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:40.973174 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:40.772466 2005 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.768541 2004 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:40.979818 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 184333us
I20260805 22:24:40.973896 31707 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:40.982205 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:40.984018 1858 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:40.972821 1478 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:40.992612 1478 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:40.997251 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 248093us
I20260805 22:24:40.999790 31444 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:41.001784 31444 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:40.815552 1797 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.817234 1803 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:41.008868 31707 raft_consensus.cc:1226] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:40.965629 1986 raft_consensus.cc:2781] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:40.951725 1545 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:40.956473 1839 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:40.916580 2018 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:40.969147 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:41.013248 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:41.013998 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:41.017005 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:41.012655 31444 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:40.965131 2052 raft_consensus.cc:2726] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:40.995255 1925 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:40.999454 2026 raft_consensus.cc:2726] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:41.010341 1478 raft_consensus.cc:2726] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:40.964048 2007 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:40.986802 2033 raft_consensus.cc:686] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:41.012864 2020 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:41.012152 1131 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:41.052263 31603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59028 (request call id 2018) took 1098 ms. Trace:
I20260805 22:24:41.051930 31499 raft_consensus.cc:2354] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4.
I20260805 22:24:41.052083 31707 raft_consensus.cc:1226] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260805 22:24:41.055655 31602 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:40.968495 2011 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:41.076830 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:41.077620 31604 raft_consensus.cc:2379] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:41.073176 31603 rpcz_store.cc:276] 0805 22:24:39.945846 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:39.946328 (+ 482us) service_pool.cc:227] Handling call
0805 22:24:41.044066 (+1097738us) spinlock_profiling.cc:243] Waited 277 ms on lock 0x7b54000b1b54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:41.044223 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":636249472}
I20260805 22:24:41.082788 2073 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:41.085469 31447 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
W20260805 22:24:41.092584 2069 log.cc:924] Time spent T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.058s user 0.001s sys 0.001s
I20260805 22:24:41.080004 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:41.072395 2007 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:41.056339 1986 raft_consensus.cc:686] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
W20260805 22:24:41.086689 2072 log.cc:924] Time spent T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.050s user 0.001s sys 0.000s
I20260805 22:24:41.054913 1485 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:41.069725 2061 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:41.099143 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:41.114024 31603 raft_consensus.cc:2379] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 3: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:41.075313 2023 consensus_queue.cc:237] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:41.099596 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:41.115321 31609 raft_consensus.cc:2379] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 4: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:41.078329 2026 raft_consensus.cc:2726] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:41.088737 31597 raft_consensus.cc:2379] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 3: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:41.033088 1545 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:41.039960 1839 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:41.101249 2012 consensus_meta.cc:287] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.766s user 0.011s sys 0.003s
I20260805 22:24:41.087877 1919 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:41.086114 2073 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:41.090862 1909 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:41.080981 1131 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:41.085129 2020 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:41.112664 1999 raft_consensus.cc:2726] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:41.118044 1485 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
W20260805 22:24:41.123016 1831 consensus_meta.cc:287] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.523s user 0.006s sys 0.003s
I20260805 22:24:41.130959 2033 consensus_queue.cc:237] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:41.104314 2007 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:41.147912 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:41.148633 31501 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 4
I20260805 22:24:41.150189 1986 consensus_queue.cc:237] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:41.151684 31499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40740 (request call id 1052) took 1263 ms. Trace:
I20260805 22:24:41.127470 1994 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:41.152062 31499 rpcz_store.cc:276] 0805 22:24:39.887887 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:39.887998 (+ 111us) service_pool.cc:227] Handling call
0805 22:24:41.151479 (+1263481us) spinlock_profiling.cc:243] Waited 1.16 s on lock 0x7b54002625d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:41.151625 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2660854528}
I20260805 22:24:41.152910 1966 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:41.153908 1797 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:41.154397 2056 log.cc:924] Time spent T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.122s user 0.001s sys 0.000s
I20260805 22:24:41.156539 1831 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:41.157349 2007 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:41.157433 1831 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:41.126734 2012 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:41.159119 1131 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:41.159919 2005 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:41.153201 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:41.162431 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:41.163008 31499 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:24:41.163793 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:41.164314 31499 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:41.165035 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:41.058118 1925 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:41.170586 31501 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:41.170501 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:41.171679 31499 raft_consensus.cc:2379] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 3: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:41.173578 2007 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:41.174412 2007 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:41.179590 1925 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:41.180681 31711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1637) took 1104 ms. Trace:
I20260805 22:24:41.181051 1839 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:41.181308 31704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1629) took 1807 ms. Trace:
I20260805 22:24:39.880512 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:41.191982 31554 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:41.137511 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:41.202204 31494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52098 (request call id 249) took 1423 ms. Trace:
I20260805 22:24:41.212688 31494 rpcz_store.cc:276] 0805 22:24:39.778685 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:39.778792 (+ 107us) service_pool.cc:227] Handling call
0805 22:24:41.201975 (+1423183us) spinlock_profiling.cc:243] Waited 1.14 s on lock 0x7b54002e5254. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:41.202139 (+ 164us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2630920192}
I20260805 22:24:41.184937 31711 rpcz_store.cc:276] 0805 22:24:40.076656 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:40.076768 (+ 112us) service_pool.cc:227] Handling call
0805 22:24:40.090117 (+ 13349us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:40.757403 (+667286us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:41.180384 (+422981us) spinlock_profiling.cc:243] Waited 617 ms on lock 0x7b54002b39d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:41.180556 (+ 172us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1419783424}
I20260805 22:24:41.204766 31704 rpcz_store.cc:276] 0805 22:24:39.374367 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:39.379047 (+ 4680us) service_pool.cc:227] Handling call
0805 22:24:39.413343 (+ 34296us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:40.165679 (+752336us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:41.181091 (+1015412us) spinlock_profiling.cc:243] Waited 752 ms on lock 0x7b5400219b54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:41.181251 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1728481024}
I20260805 22:24:41.151904 2018 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:41.179450 2073 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:41.177266 2007 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:41.160893 2012 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:41.220885 31554 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:41.183032 2004 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:41.140592 1485 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:41.228217 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:41.229127 1831 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:41.232893 31494 raft_consensus.cc:1261] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260805 22:24:41.123303 2061 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:41.235148 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 140325us
I20260805 22:24:41.249157 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:41.249497 2011 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:41.254698 2020 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:24:41.258033 1958 consensus_meta.cc:287] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.631s user 0.013s sys 0.000s
I20260805 22:24:41.258657 1545 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:41.239790 2018 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:41.259589 31596 raft_consensus.cc:2396] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 3 because replica has last-logged OpId of term: 2 index: 2, which is greater than that of the candidate, which has last-logged OpId of term: 1 index: 1.
I20260805 22:24:41.237416 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:24:41.226665 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 128439us
I20260805 22:24:41.241114 31704 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:41.211053 1803 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:41.220261 2075 raft_consensus.cc:2726] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:41.270910 31707 raft_consensus.cc:2379] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 3: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:41.272338 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:41.264925 31447 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:41.273113 31603 raft_consensus.cc:2379] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:41.274042 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:41.259482 2018 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:41.287741 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:41.286031 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:41.291602 31554 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:41.292861 31604 raft_consensus.cc:2379] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 4: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
W20260805 22:24:41.279028 2030 consensus_meta.cc:287] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.510s user 0.008s sys 0.003s
I20260805 22:24:41.241089 1753 consensus_queue.cc:1046] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.047s
I20260805 22:24:41.317255 31594 raft_consensus.cc:2396] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260805 22:24:41.321959 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:41.325909 31604 raft_consensus.cc:2379] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 4: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:41.327647 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:41.290652 2018 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:41.331403 31711 raft_consensus.cc:1261] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260805 22:24:41.297945 1765 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:41.289621 1803 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:41.345235 1949 raft_consensus.cc:3032] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:41.345754 1803 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:41.266425 1741 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:41.344376 2032 raft_consensus.cc:3032] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:41.279403 1780 consensus_queue.cc:1035] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.050s
W20260805 22:24:41.325290 1927 consensus_meta.cc:287] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.527s user 0.010s sys 0.002s
I20260805 22:24:41.355206 31554 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:41.365571 31499 raft_consensus.cc:1226] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:41.345121 1765 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:41.370008 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:41.370185 1765 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
W20260805 22:24:41.371449 1888 consensus_meta.cc:287] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.607s user 0.008s sys 0.004s
I20260805 22:24:41.372167 31600 raft_consensus.cc:1261] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260805 22:24:41.373857 1803 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:41.345798 1949 raft_consensus.cc:729] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:41.376643 31715 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:41.358180 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:41.341039 1574 consensus_meta.cc:287] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.953s user 0.013s sys 0.000s
I20260805 22:24:41.365671 1927 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:41.378680 1485 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:41.380162 31709 raft_consensus.cc:2354] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4.
I20260805 22:24:41.379544 1927 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:41.380847 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:41.382112 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:41.383037 1741 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:41.383868 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:41.386389 31494 raft_consensus.cc:1226] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:41.387435 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:41.369338 31499 raft_consensus.cc:1261] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260805 22:24:41.376662 1949 consensus_queue.cc:260] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:41.387635 31494 raft_consensus.cc:1226] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:41.389964 31501 raft_consensus.cc:2379] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:41.368806 31604 raft_consensus.cc:3032] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:41.391870 31604 raft_consensus.cc:729] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:41.395094 1765 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:41.395819 1951 consensus_meta.cc:287] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.602s user 0.006s sys 0.004s
I20260805 22:24:41.357514 2032 raft_consensus.cc:729] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:41.401108 31705 raft_consensus.cc:2396] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 4 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260805 22:24:41.401547 2032 consensus_queue.cc:260] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:41.403293 2032 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:41.408836 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:41.410044 32583 raft_consensus.cc:2354] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:24:41.409952 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:41.411216 2070 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:41.411365 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:41.412629 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:41.396396 31604 consensus_queue.cc:260] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:41.413826 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 58702.5us
I20260805 22:24:41.411459 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:41.379451 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:41.391232 2061 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:41.368050 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 251721us
I20260805 22:24:41.416190 31704 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:41.418242 31667 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:41.385128 31490 raft_consensus.cc:1261] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260805 22:24:41.420635 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 132496us
I20260805 22:24:41.416630 1574 raft_consensus.cc:2726] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:41.422947 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:41.423714 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:41.425774 31447 leader_election.cc:398] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Vote denied by peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801) with higher term. Message: Invalid argument: T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
W20260805 22:24:41.427796 32529 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.194:41897 timed out after 1.548s (SENT) [suppressed 36 similar messages]
I20260805 22:24:41.438810 31447 leader_election.cc:401] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260805 22:24:41.440335 31447 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:41.438338 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:41.441210 31601 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:41.442541 31554 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:41.442533 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:41.443691 31494 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:41.444410 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:41.421655 1951 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:41.442842 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:41.452561 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:41.448762 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:41.421576 32584 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57366 (request call id 345) took 1543 ms (client timeout 1840 ms). Trace:
I20260805 22:24:41.453294 31602 raft_consensus.cc:2379] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:41.449456 31499 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:41.451752 31704 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:41.454315 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:41.456347 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:41.453359 31596 raft_consensus.cc:1261] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 4. (term mismatch)
I20260805 22:24:41.452675 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:41.457176 31596 pending_rounds.cc:76] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c: Aborting all ops after (but not including) 3
I20260805 22:24:41.457489 31596 pending_rounds.cc:97] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c: Aborting uncommitted NO_OP operation due to leader change: 5.4
I20260805 22:24:41.457497 31609 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:41.457968 31596 raft_consensus.cc:2866] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:24:41.461149 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:41.462772 31501 raft_consensus.cc:2379] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:41.464042 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:41.441870 1982 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:41.448076 1951 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:41.476451 31554 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:41.477123 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:41.429195 1927 consensus_queue.cc:1046] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260805 22:24:41.452394 1803 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:41.480803 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:41.482489 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:41.459326 1952 raft_consensus.cc:2726] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:41.488874 1574 raft_consensus.cc:2781] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:41.490046 1765 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:41.499111 31602 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 4
I20260805 22:24:41.486497 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:41.483563 31596 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:41.498483 31609 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:24:41.503418 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:41.438908 2031 consensus_queue.cc:1046] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260805 22:24:41.509267 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:41.481884 1951 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:41.465734 1988 consensus_queue.cc:1035] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.041s
I20260805 22:24:41.534421 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
W20260805 22:24:41.523541 31594 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56384 (request call id 257) took 1292 ms (client timeout 1548 ms). Trace:
I20260805 22:24:41.535249 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:24:41.455663 32584 rpcz_store.cc:269] 0805 22:24:39.877900 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:39.888469 (+ 10569us) service_pool.cc:227] Handling call
0805 22:24:41.421140 (+1532671us) spinlock_profiling.cc:243] Waited 351 ms on lock 0x7b54004f51d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:41.421401 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":806571264}
I20260805 22:24:41.478655 2075 consensus_queue.cc:1035] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.043s
I20260805 22:24:41.479274 2030 consensus_queue.cc:1046] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.039s
I20260805 22:24:41.516647 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:41.540321 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:41.703557 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:41.915481 1765 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:41.931183 31494 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:24:41.724228 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 296423us
I20260805 22:24:41.902940 2070 raft_consensus.cc:2726] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:41.933276 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:41.841094 31597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59028 (request call id 2028) took 1598 ms. Trace:
W20260805 22:24:41.595064 2047 log.cc:924] Time spent T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.896s user 0.002s sys 0.004s
W20260805 22:24:41.571980 32579 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57382 (request call id 542) took 1316 ms (client timeout 1722 ms). Trace:
I20260805 22:24:41.936829 31714 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58348 (request call id 372) took 1664 ms. Trace:
I20260805 22:24:41.598079 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
W20260805 22:24:41.535621 31594 rpcz_store.cc:269] 0805 22:24:40.231444 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:40.235794 (+ 4350us) service_pool.cc:227] Handling call
0805 22:24:41.224788 (+988994us) spinlock_profiling.cc:243] Waited 2.84 ms on lock 0x7b5800004598. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735faa1b0 00007f2735fa9a94 00007f272f691c04 00007f272f691b92
0805 22:24:41.522043 (+297255us) spinlock_profiling.cc:243] Waited 66.9 ms on lock 0x7b54003bf6d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:41.523466 (+ 1423us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":160472704}
I20260805 22:24:41.937706 1988 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:41.952401 2052 consensus_queue.cc:1035] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20260805 22:24:41.936522 32579 rpcz_store.cc:269] 0805 22:24:40.255352 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:40.257305 (+ 1953us) service_pool.cc:227] Handling call
0805 22:24:41.571655 (+1314350us) spinlock_profiling.cc:243] Waited 463 ms on lock 0x7b5400652054. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:41.571866 (+ 211us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1064566656}
I20260805 22:24:41.546314 1905 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:41.937479 31714 rpcz_store.cc:276] 0805 22:24:40.272639 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:40.282544 (+ 9905us) service_pool.cc:227] Handling call
0805 22:24:40.298094 (+ 15550us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:40.301784 (+ 3690us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:40.301812 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:40.306306 (+ 4494us) log.cc:842] Serialized 26 byte log entry
0805 22:24:40.533737 (+227431us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:24:40.533777 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:40.533846 (+ 69us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:41.594218 (+1060372us) spinlock_profiling.cc:243] Waited 32 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:24:41.936539 (+342321us) raft_consensus.cc:1671] finished
0805 22:24:41.936560 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:41.936770 (+ 210us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":98,"raft.run_cpu_time_us":125,"raft.run_wall_time_us":4024,"spinlock_wait_cycles":73537408,"thread_start_us":159170,"threads_started":1,"wal-append.queue_time_us":271087}
I20260805 22:24:41.954381 1905 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:41.955323 1988 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:41.956056 31713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 2073) took 1654 ms. Trace:
I20260805 22:24:41.842496 31603 raft_consensus.cc:1261] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260805 22:24:41.843312 31493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40740 (request call id 1056) took 1570 ms. Trace:
I20260805 22:24:41.957589 2067 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:41.958204 2067 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:41.959889 31706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56260 (request call id 2066) took 1972 ms (client timeout 1633 ms). Trace:
I20260805 22:24:41.960189 2067 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:41.562235 2097 log.cc:924] Time spent T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.083s user 0.000s sys 0.000s
I20260805 22:24:41.801486 1975 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:41.938192 2085 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:41.552968 2018 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:41.931262 1765 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:41.941846 31500 raft_consensus.cc:2379] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:41.956590 31713 rpcz_store.cc:276] 0805 22:24:40.301578 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:40.593767 (+292189us) service_pool.cc:227] Handling call
0805 22:24:40.620338 (+ 26571us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:40.708913 (+ 88575us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:40.708941 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:40.708979 (+ 38us) raft_consensus.cc:1631] Marking committed up to 2
0805 22:24:40.709012 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:41.955831 (+1246819us) spinlock_profiling.cc:243] Waited 88.4 ms on lock 0x7b5400151b54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:41.955879 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:41.955999 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":203246848}
I20260805 22:24:41.930286 31609 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:41.979197 31494 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:41.976241 2070 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:41.981395 2070 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:41.958732 1988 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:41.639679 31602 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
W20260805 22:24:41.960315 31706 rpcz_store.cc:269] 0805 22:24:39.987401 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:39.987970 (+ 569us) service_pool.cc:227] Handling call
0805 22:24:41.959667 (+1971697us) spinlock_profiling.cc:243] Waited 744 ms on lock 0x7b540016d854. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:41.959798 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1710393728}
I20260805 22:24:41.913368 1982 raft_consensus.cc:2726] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801) with higher term. Message: Invalid argument: T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:24:41.957007 31493 rpcz_store.cc:276] 0805 22:24:40.272621 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:40.381824 (+109203us) service_pool.cc:227] Handling call
0805 22:24:40.382568 (+ 744us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:40.393794 (+ 11226us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:40.393824 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:40.394794 (+ 970us) log.cc:842] Serialized 26 byte log entry
0805 22:24:41.412827 (+1018033us) spinlock_profiling.cc:243] Waited 9.61 ms on lock 0x7b180014d460. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e122345 00007f2736f59a39 00007f272e13f8bb 00007f272e13dcbf 00007f272e13fe22 00007f2730ce95bb 00007f2730cef063 00007f2730cef411 00007f273103c5cf
0805 22:24:41.456827 (+ 44000us) raft_consensus.cc:1631] Marking committed up to 3
0805 22:24:41.456869 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:41.456942 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:41.842939 (+385997us) raft_consensus.cc:1671] finished
0805 22:24:41.842964 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:41.843242 (+ 278us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9309,"raft.queue_time_us":6810,"raft.run_cpu_time_us":132,"raft.run_wall_time_us":3713,"spinlock_wait_cycles":22113024,"thread_start_us":43801,"threads_started":1,"wal-append.queue_time_us":1379508}
I20260805 22:24:41.936832 1478 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:41.989804 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:41.991206 32529 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:41.988435 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:41.934015 31597 rpcz_store.cc:276] 0805 22:24:40.242588 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:40.243970 (+ 1382us) service_pool.cc:227] Handling call
0805 22:24:41.840781 (+1596811us) spinlock_profiling.cc:243] Waited 747 ms on lock 0x7b5400237dd4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:41.841007 (+ 226us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1717553920}
I20260805 22:24:41.914883 1951 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:41.985296 2067 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:42.002465 32529 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:41.997891 1838 raft_consensus.cc:2781] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:41.991961 2078 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:41.983822 1975 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.076920 31711 raft_consensus.cc:2354] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 6. Current term is 7.
I20260805 22:24:42.076840 31501 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:42.062188 1905 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:41.983137 2085 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:42.088258 32591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57372 (request call id 542) took 1550 ms. Trace:
I20260805 22:24:41.993230 1478 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:42.089543 32591 rpcz_store.cc:276] 0805 22:24:40.538068 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:40.538223 (+ 155us) service_pool.cc:227] Handling call
0805 22:24:42.088069 (+1549846us) spinlock_profiling.cc:243] Waited 4.06 ms on lock 0x7b54004303d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:42.088193 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":9329408}
I20260805 22:24:42.075908 2016 raft_consensus.cc:2726] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260805 22:24:42.093065 1574 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.009s sys 0.009s
W20260805 22:24:42.095976 1888 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.589s user 0.001s sys 0.009s
I20260805 22:24:42.063407 2080 consensus_queue.cc:1035] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.114s
I20260805 22:24:42.095747 1931 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:42.065366 2067 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:42.129567 1888 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.623s user 0.002s sys 0.010s
I20260805 22:24:42.117087 1902 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:42.079488 2078 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.140218 31609 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
W20260805 22:24:42.125860 1574 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.586s user 0.010s sys 0.009s
I20260805 22:24:42.144032 2070 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:42.171299 1886 raft_consensus.cc:3032] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:42.175011 2018 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:42.145783 1931 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.184515 1886 raft_consensus.cc:729] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:42.170719 1902 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:42.188923 2067 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:42.200757 31554 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.700s user 0.015s sys 0.017s
I20260805 22:24:42.190224 1478 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:42.202409 31554 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.702s user 0.015s sys 0.017s
I20260805 22:24:42.204077 1975 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:42.201723 2067 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:24:42.206918 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 706364us
I20260805 22:24:42.205622 2018 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:42.206598 2067 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:42.201930 1982 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:42.209371 2018 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.209613 1982 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:42.210847 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:42.211359 1982 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:42.211601 32594 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57372 (request call id 545) took 1254 ms (client timeout 1569 ms). Trace:
W20260805 22:24:42.211979 32594 rpcz_store.cc:269] 0805 22:24:40.957136 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:40.957259 (+ 123us) service_pool.cc:227] Handling call
0805 22:24:42.210682 (+1253423us) spinlock_profiling.cc:243] Waited 1.03 ms on lock 0x7b5800005198. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735faa1b0 00007f2735fa9a94 00007f272f691c04 00007f272f691b92
0805 22:24:42.211546 (+ 864us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2377984}
I20260805 22:24:42.212740 1958 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:42.212980 2067 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:42.208513 2004 raft_consensus.cc:3032] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:42.222373 31499 raft_consensus.cc:1226] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260805 22:24:42.195892 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:42.227406 31704 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.691s user 0.000s sys 0.050s
W20260805 22:24:42.227825 31704 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.691s user 0.000s sys 0.050s
W20260805 22:24:42.231104 31668 thread.cc:636] dns-resolver [worker] (thread pool) Time spent creating pthread: real 0.508s user 0.012s sys 0.020s
W20260805 22:24:42.231465 31668 thread.cc:602] dns-resolver [worker] (thread pool) Time spent starting thread: real 0.509s user 0.012s sys 0.020s
W20260805 22:24:42.233129 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 514646us
I20260805 22:24:42.170370 1574 raft_consensus.cc:686] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:42.211416 2018 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:42.235813 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:42.236766 31493 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 5
I20260805 22:24:42.238433 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:42.239068 32594 raft_consensus.cc:2454] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:42.213483 2004 raft_consensus.cc:729] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:42.197067 1765 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:42.295161 31668 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:42.296590 31706 raft_consensus.cc:1226] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:42.299796 2096 raft_consensus.cc:2781] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:42.300226 2096 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:42.300580 2096 raft_consensus.cc:3037] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:42.301153 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:42.301851 31714 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:42.305814 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:42.306341 2116 log.cc:924] Time spent T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260805 22:24:42.306535 31713 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
W20260805 22:24:42.309821 2107 log.cc:924] Time spent T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
I20260805 22:24:42.310285 1888 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:42.311136 31708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58348 (request call id 374) took 2010 ms. Trace:
I20260805 22:24:42.310856 1888 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.311497 31708 rpcz_store.cc:276] 0805 22:24:40.301107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:40.590388 (+289281us) service_pool.cc:227] Handling call
0805 22:24:40.600718 (+ 10330us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:40.601119 (+ 401us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:40.601154 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:41.423025 (+821871us) spinlock_profiling.cc:243] Waited 11.2 ms on lock 0x7b2c00004908. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272fa4be11 00007f272fa5b442 00007f27310a3c11 00007f273106858a 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772
0805 22:24:41.423422 (+ 397us) log.cc:842] Serialized 26 byte log entry
0805 22:24:41.636692 (+213270us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:24:41.636731 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:41.636807 (+ 76us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:42.310854 (+674047us) raft_consensus.cc:1671] finished
0805 22:24:42.310886 (+ 32us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:42.311081 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":500,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"spinlock_wait_cycles":25685888,"thread_start_us":212855,"threads_started":1,"wal-append.queue_time_us":707356}
I20260805 22:24:42.231549 2085 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:42.318219 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:42.322820 2099 log.cc:924] Time spent T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.761s user 0.010s sys 0.000s
I20260805 22:24:42.293465 1754 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:42.222920 1958 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.328222 31714 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:42.213771 1886 consensus_queue.cc:260] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.327592 1951 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:42.329672 2078 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:42.326407 1765 consensus_queue.cc:1046] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.043s
I20260805 22:24:42.363584 1975 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:42.331851 1754 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:42.377766 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 438693us
W20260805 22:24:42.386636 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 76757.4us
I20260805 22:24:42.390388 1888 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:42.390936 1754 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:42.391168 31554 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:42.391672 1949 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:42.391952 1888 raft_consensus.cc:2781] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:42.392356 1888 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:42.392661 1888 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
W20260805 22:24:42.392702 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 390267us
I20260805 22:24:42.383749 2085 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:42.394958 31554 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:42.394194 31495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.GetConsensusState from 127.0.0.1:40714 (request call id 1423) took 2097 ms. Trace:
I20260805 22:24:42.398082 31495 rpcz_store.cc:276] 0805 22:24:40.296336 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:40.417564 (+121228us) service_pool.cc:227] Handling call
0805 22:24:42.393749 (+1976185us) spinlock_profiling.cc:243] Waited 569 ms on lock 0x7b54000c17d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fedaf0 00007f2731070f2a 00007f2735fadc6e 00007f272f6a4807 00007f272f6a4792 00007f272f6a46c1 00007f272f6a463c 00007f272f6a38f2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:42.394131 (+ 382us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1309088000}
I20260805 22:24:42.400664 2085 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.401690 1986 consensus_queue.cc:1035] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260805 22:24:42.375334 1931 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:42.403463 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.419561 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.382691 1902 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:42.421015 31705 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1633) took 2358 ms (client timeout 2646 ms). Trace:
I20260805 22:24:42.422616 2078 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:42.369737 1574 consensus_queue.cc:237] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:42.409114 2032 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:42.424185 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:42.435400 31501 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:42.425628 1958 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:42.425649 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:42.437781 31554 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
W20260805 22:24:42.426195 31709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1654) took 1821 ms (client timeout 2208 ms). Trace:
I20260805 22:24:42.361008 2004 consensus_queue.cc:260] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:42.427794 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:42.440979 31554 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:42.371516 1975 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:42.417711 31714 raft_consensus.cc:2379] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 3: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:42.428185 31493 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:42.429054 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:42.429718 32529 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:42.436226 2078 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:42.433030 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:42.438416 31709 rpcz_store.cc:269] 0805 22:24:40.605021 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:40.741665 (+136644us) service_pool.cc:227] Handling call
0805 22:24:42.426004 (+1684339us) spinlock_profiling.cc:243] Waited 621 ms on lock 0x7b540016c954. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:42.426136 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1429073152}
I20260805 22:24:42.426863 31486 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40740 (request call id 1060) took 2041 ms. Trace:
I20260805 22:24:42.506923 31486 rpcz_store.cc:276] 0805 22:24:40.385850 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:40.416019 (+ 30169us) service_pool.cc:227] Handling call
0805 22:24:40.416300 (+ 281us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:40.949396 (+533096us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:40.954568 (+ 5172us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:40.965232 (+ 10664us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:24:40.969287 (+ 4055us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:42.426637 (+1457350us) spinlock_profiling.cc:243] Waited 472 ms on lock 0x7b54000c17d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:42.426682 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:42.426806 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1086634752}
I20260805 22:24:42.440986 31501 raft_consensus.cc:1226] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:42.508626 1975 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:42.482244 31492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1597) took 1875 ms. Trace:
I20260805 22:24:42.515036 31359 catalog_manager.cc:5495] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:42.516919 31359 catalog_manager.cc:5495] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
W20260805 22:24:42.484766 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 137145us
I20260805 22:24:42.475126 32579 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:42.439338 2032 raft_consensus.cc:686] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
W20260805 22:24:42.434110 31705 rpcz_store.cc:269] 0805 22:24:40.062428 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:40.072692 (+ 10264us) service_pool.cc:227] Handling call
0805 22:24:42.420780 (+2348088us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b5400315454. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:42.420931 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3003120896}
I20260805 22:24:42.531775 31554 leader_election.cc:398] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4.
I20260805 22:24:42.532091 31554 leader_election.cc:401] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260805 22:24:42.480856 1765 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:42.533258 1754 raft_consensus.cc:2726] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4.
I20260805 22:24:42.533469 31554 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:42.535899 31668 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:42.487819 31495 raft_consensus.cc:1226] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:42.445850 1958 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:42.432337 1913 raft_consensus.cc:3032] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:42.485800 1858 raft_consensus.cc:2726] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:42.549739 1913 raft_consensus.cc:729] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:42.476883 1856 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:42.435248 1902 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:42.522722 31490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40740 (request call id 1070) took 1080 ms. Trace:
W20260805 22:24:42.523717 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 94075.4us
I20260805 22:24:42.524804 2029 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:42.531852 2115 consensus_queue.cc:1046] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.003s
I20260805 22:24:42.542279 2010 raft_consensus.cc:2726] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:42.545940 31351 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47946 (request call id 184) took 4832 ms. Trace:
I20260805 22:24:42.547644 31581 tablet_service.cc:1514] Processing DeleteTablet for tablet e6cf90ae3fc840ceb48be8c790be97d9 with delete_type TABLET_DATA_TOMBSTONED (TS 516cb07ce3064ccbbca35e912d62816c not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:59024
I20260805 22:24:42.487493 2004 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:42.504537 2096 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.553516 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:42.554291 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:42.557101 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:42.557583 31603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 2046) took 1282 ms. Trace:
I20260805 22:24:42.550345 1856 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.558712 2032 consensus_queue.cc:237] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:42.514460 31492 rpcz_store.cc:276] 0805 22:24:40.607219 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:40.818331 (+211112us) service_pool.cc:227] Handling call
0805 22:24:40.945741 (+127410us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:40.946127 (+ 386us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:40.946155 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:40.946649 (+ 494us) log.cc:842] Serialized 26 byte log entry
0805 22:24:41.155473 (+208824us) spinlock_profiling.cc:243] Waited 1.79 ms on lock 0x7b18002260c0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e122345 00007f2736f59a39 00007f272e13f8bb 00007f272e13dcbf 00007f272e13fe22 00007f2730ce95bb 00007f2730cef063 00007f2730cef411 00007f273103c5cf
0805 22:24:41.286848 (+131375us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:24:41.496329 (+209481us) spinlock_profiling.cc:243] Waited 1.97 ms on lock 0x7b18002260c0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e140638 00007f272e143b11 00007f272e143ac9 00007f272e143a59 00007f272e143a21 00007f272e142d1d 00007f27373a4204 00007f27373a4039 00007f272e122dc6
0805 22:24:42.441884 (+945555us) spinlock_profiling.cc:243] Waited 30.1 ms on lock 0x7b18002260c0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f2730cf782c 00007f272e1229dc 00007f2736f59a39 00007f272e13f8bb 00007f272e13dcbf 00007f272e13d155 00007f2734a3d167 00007f2731072807 00007f2731081f0f
0805 22:24:42.442027 (+ 143us) log.cc:842] Serialized 14 byte log entry
0805 22:24:42.481904 (+ 39877us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:42.481994 (+ 90us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:42.482035 (+ 41us) raft_consensus.cc:1671] finished
0805 22:24:42.482046 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:42.482191 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":14456,"prepare.queue_time_us":37180,"prepare.run_cpu_time_us":51,"prepare.run_wall_time_us":51,"raft.queue_time_us":52605,"raft.run_cpu_time_us":357,"raft.run_wall_time_us":26196,"spinlock_wait_cycles":77907968,"thread_start_us":203165,"threads_started":2,"wal-append.queue_time_us":436067}
I20260805 22:24:42.560135 31359 catalog_manager.cc:5495] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 4 to 3, leader changed from <none> to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 3 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:42.571650 31603 rpcz_store.cc:276] 0805 22:24:41.275529 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:41.279439 (+ 3910us) service_pool.cc:227] Handling call
0805 22:24:41.842093 (+562654us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:41.957202 (+115109us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:42.557358 (+600156us) spinlock_profiling.cc:243] Waited 562 ms on lock 0x7b5400237dd0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731067a09 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:42.557527 (+ 169us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1293143424}
I20260805 22:24:42.561209 1902 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:42.581424 31609 raft_consensus.cc:2379] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:42.555752 31490 rpcz_store.cc:276] 0805 22:24:41.442708 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:41.937263 (+494555us) service_pool.cc:227] Handling call
0805 22:24:41.937613 (+ 350us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:41.980733 (+ 43120us) raft_consensus.cc:1520] Early marking committed up to index 3
0805 22:24:41.996159 (+ 15426us) log.cc:842] Serialized 14 byte log entry
0805 22:24:42.364100 (+367941us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:42.365865 (+ 1765us) raft_consensus.cc:1631] Marking committed up to 3
0805 22:24:42.368731 (+ 2866us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:42.512045 (+143314us) spinlock_profiling.cc:243] Waited 999 us on lock 0x7b18003978a0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e10df96 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9
0805 22:24:42.513219 (+ 1174us) spinlock_profiling.cc:243] Waited 19.4 ms on lock 0x7b54003a2854. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:42.513268 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:42.522651 (+ 9383us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":1698,"prepare.run_cpu_time_us":38,"prepare.run_wall_time_us":40,"raft.queue_time_us":2055,"raft.run_cpu_time_us":115,"raft.run_wall_time_us":3058,"spinlock_wait_cycles":46942848,"thread_start_us":279988,"threads_started":1,"wal-append.queue_time_us":418290}
I20260805 22:24:42.589475 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:42.544574 1958 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:42.565930 1975 raft_consensus.cc:2781] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:42.594913 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:42.595467 2029 raft_consensus.cc:686] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:42.596334 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:42.555470 2130 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:42.566154 1998 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:42.562654 31351 rpcz_store.cc:276] 0805 22:24:37.694530 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:37.694769 (+ 239us) service_pool.cc:227] Handling call
0805 22:24:42.527186 (+4832417us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0805 22:24:39.300894 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0805 22:24:39.327611 (+ 26717us) write_op.cc:432] Acquiring schema lock in shared mode
0805 22:24:39.327651 (+ 40us) write_op.cc:435] Acquired schema lock
0805 22:24:39.327662 (+ 11us) tablet.cc:660] Decoding operations
0805 22:24:39.328222 (+ 560us) write_op.cc:620] Acquiring the partition lock for write op
0805 22:24:39.328282 (+ 60us) write_op.cc:641] Partition lock acquired for write op
0805 22:24:39.328309 (+ 27us) tablet.cc:683] Acquiring locks for 4 operations
0805 22:24:39.328699 (+ 390us) tablet.cc:699] Row locks acquired
0805 22:24:39.328714 (+ 15us) write_op.cc:260] PREPARE: finished
0805 22:24:39.328860 (+ 146us) write_op.cc:270] Start()
0805 22:24:39.328977 (+ 117us) write_op.cc:276] Timestamp: P: 1785968679328815 usec, L: 0
0805 22:24:39.329031 (+ 54us) op_driver.cc:331] REPLICATION: starting
0805 22:24:39.393624 (+ 64593us) log.cc:842] Serialized 1652 byte log entry
0805 22:24:39.468868 (+ 75244us) op_driver.cc:444] REPLICATION: finished
0805 22:24:39.641981 (+173113us) write_op.cc:301] APPLY: starting
0805 22:24:39.642072 (+ 91us) tablet.cc:1365] starting BulkCheckPresence
0805 22:24:39.678644 (+ 36572us) tablet.cc:1368] finished BulkCheckPresence
0805 22:24:39.678778 (+ 134us) tablet.cc:1370] starting ApplyRowOperation cycle
0805 22:24:39.679403 (+ 625us) tablet.cc:1381] finished ApplyRowOperation cycle
0805 22:24:39.679654 (+ 251us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0805 22:24:39.679706 (+ 52us) write_op.cc:312] APPLY: finished
0805 22:24:39.679942 (+ 236us) log.cc:842] Serialized 41 byte log entry
0805 22:24:39.680134 (+ 192us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:39.680237 (+ 103us) write_op.cc:454] Released schema lock
0805 22:24:39.680282 (+ 45us) write_op.cc:341] FINISH: Updating metrics
0805 22:24:39.680606 (+ 324us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:39.680633 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":998,"reported_tablets":9,"child_traces":[["op",{"apply.queue_time_us":163163,"apply.run_cpu_time_us":2587,"apply.run_wall_time_us":38953,"num_ops":4,"prepare.queue_time_us":228376,"prepare.run_cpu_time_us":3866,"prepare.run_wall_time_us":119103,"replication_time_us":139726,"thread_start_us":244087,"threads_started":2}]]}
I20260805 22:24:42.603767 31501 raft_consensus.cc:1226] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260805 22:24:42.604023 32529 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:42.607602 31359 catalog_manager.cc:5495] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:24:42.591164 1754 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:42.594056 1958 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:42.614782 32529 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:42.600363 1975 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:42.605907 1803 consensus_queue.cc:1035] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.025s
I20260805 22:24:42.632380 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:42.635607 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.625810 2022 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:42.610328 2130 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.637741 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:42.637741 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.638453 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.638967 31705 raft_consensus.cc:2354] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5.
I20260805 22:24:42.613597 31600 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:42.639541 2096 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:42.610291 31597 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:42.631510 1975 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:42.612371 1998 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:42.626080 2080 raft_consensus.cc:2726] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:42.642591 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:42.638496 31706 raft_consensus.cc:2379] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 4: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:42.643687 31444 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:42.636442 31708 raft_consensus.cc:1261] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 3 index: 2. (index mismatch)
I20260805 22:24:42.652086 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.652686 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.654052 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.654181 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.626816 1754 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.655761 31444 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:42.656406 2130 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:42.656569 1803 raft_consensus.cc:2781] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:42.657015 1803 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:42.657425 1803 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:42.658004 2064 raft_consensus.cc:2781] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:42.658391 2064 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:42.659116 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.639097 31709 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:42.663972 31359 catalog_manager.cc:5495] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:42.666365 1902 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:42.665838 31359 catalog_manager.cc:5495] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 3 to 5, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 5 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:42.667412 31359 catalog_manager.cc:5495] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 1 to 3, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 3 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:24:42.669095 31556 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:42.670181 31715 raft_consensus.cc:1261] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:24:42.670876 31556 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:42.671550 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.672124 31707 raft_consensus.cc:2454] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:42.672541 31707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56260 (request call id 2078) took 1237 ms. Trace:
I20260805 22:24:42.672884 31486 raft_consensus.cc:1261] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:24:42.655311 2140 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:42.674065 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:42.674815 31486 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:42.676741 31493 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:42.673888 2140 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.674875 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:42.655196 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:42.679653 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:42.681877 31603 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:24:42.668615 31359 catalog_manager.cc:5495] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 1 to 2, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 2 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:42.672798 31707 rpcz_store.cc:276] 0805 22:24:41.435656 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:42.634447 (+1198791us) service_pool.cc:227] Handling call
0805 22:24:42.672492 (+ 38045us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:24:42.655195 31667 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:42.658752 2064 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:42.685887 31708 raft_consensus.cc:2379] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:42.693838 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.694226 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.694842 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:42.695158 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.695758 31713 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:42.697503 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:42.694815 31707 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:42.698716 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.698385 31359 catalog_manager.cc:5495] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:42.699748 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.695648 31495 raft_consensus.cc:2379] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 4: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:42.698354 32594 raft_consensus.cc:2454] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
W20260805 22:24:42.700587 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 45478.4us
I20260805 22:24:42.697183 31554 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:42.671187 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.698071 32587 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:42.704429 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:42.704893 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:42.709683 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.705224 31667 leader_election.cc:398] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Vote denied by peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607) with higher term. Message: Invalid argument: T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4.
I20260805 22:24:42.693691 1902 raft_consensus.cc:2781] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:42.706065 1952 raft_consensus.cc:2781] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:42.706169 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:42.706733 1856 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:42.706876 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:42.710750 31667 leader_election.cc:401] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Cancelling election due to peer responding with higher term
I20260805 22:24:42.711254 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:42.676366 1920 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:42.657249 1754 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:42.681305 1915 raft_consensus.cc:2726] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:42.717293 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:42.719527 31707 raft_consensus.cc:1261] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:42.702860 2137 raft_consensus.cc:2741] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election decision vote started in defunct term 2: won
I20260805 22:24:42.721274 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.721933 31668 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:42.734257 31707 raft_consensus.cc:1261] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:24:42.661748 1765 consensus_queue.cc:1046] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20260805 22:24:42.726598 31556 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:42.741595 2075 raft_consensus.cc:2781] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:42.741291 31556 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:42.717078 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:42.727051 32529 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:42.727533 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:42.727620 31667 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:42.744987 31495 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:24:42.700345 31708 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:42.712739 1902 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:42.716037 1952 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:42.742722 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:42.747470 32529 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
W20260805 22:24:42.738580 2147 log.cc:924] Time spent T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.117s user 0.003s sys 0.002s
I20260805 22:24:42.771221 31556 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:42.730304 1951 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:42.783833 31556 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:42.773093 31708 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:24:42.733013 1950 raft_consensus.cc:2726] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: Vote denied by peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607) with higher term. Message: Invalid argument: T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4.
I20260805 22:24:42.772668 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:42.749660 2161 raft_consensus.cc:2726] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:42.721442 2033 consensus_queue.cc:1035] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260805 22:24:42.697238 2029 consensus_queue.cc:237] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.742061 2075 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:42.744436 2137 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:42.762364 1902 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:42.760730 1754 raft_consensus.cc:2726] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:42.742127 2158 raft_consensus.cc:2781] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:42.820763 2158 raft_consensus.cc:686] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:42.822052 2158 consensus_queue.cc:237] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.772034 1952 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:42.764364 1989 raft_consensus.cc:2781] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:42.782896 2133 raft_consensus.cc:2781] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:42.732860 1920 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:42.796278 2014 raft_consensus.cc:2726] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:42.782959 2076 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:24:42.843662 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 78289.6us
I20260805 22:24:42.800514 2061 raft_consensus.cc:2726] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:42.801699 1998 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:42.761812 1888 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.811270 2075 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:42.835076 2149 tablet_replica.cc:330] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c: stopping tablet replica
I20260805 22:24:42.831440 2133 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:42.865294 2149 raft_consensus.cc:2229] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus shutting down.
W20260805 22:24:42.854688 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 108852us
I20260805 22:24:42.828873 1989 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:42.873365 31667 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:42.804965 1951 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.813275 1950 consensus_queue.cc:1035] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.054s
I20260805 22:24:42.878690 31667 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:42.851596 2076 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.879545 2161 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:42.874676 2133 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:42.882297 31603 raft_consensus.cc:1226] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:42.886085 31667 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:42.881150 1913 consensus_queue.cc:260] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:42.921872 2125 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:42.877537 1989 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:42.853840 1931 consensus_queue.cc:1035] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260805 22:24:42.881064 1920 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:42.866206 1856 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
W20260805 22:24:42.929497 2153 log.cc:924] Time spent T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.153s user 0.000s sys 0.001s
I20260805 22:24:42.828754 2137 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:42.892047 2161 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:42.965497 31604 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:42.901835 2022 raft_consensus.cc:2726] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:42.937151 2018 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
W20260805 22:24:42.974709 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 180278us
W20260805 22:24:42.963327 2166 log.cc:924] Time spent T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.052s user 0.000s sys 0.000s
I20260805 22:24:42.981176 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.984018 31502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1599) took 2376 ms. Trace:
I20260805 22:24:42.986765 31502 rpcz_store.cc:276] 0805 22:24:40.607530 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:41.598488 (+990958us) service_pool.cc:227] Handling call
0805 22:24:41.935382 (+336894us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:42.429727 (+494345us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:42.429758 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:42.429796 (+ 38us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:24:42.429838 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:42.983787 (+553949us) spinlock_profiling.cc:243] Waited 494 ms on lock 0x7b540033fc54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:42.983830 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:42.983964 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1136413440}
I20260805 22:24:42.952543 2096 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:42.993080 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:42.969693 2161 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:42.998076 31556 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:42.991346 1949 consensus_meta.cc:287] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.599s user 0.013s sys 0.000s
I20260805 22:24:42.979899 2032 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:42.984898 1913 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:42.942245 2125 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:42.984889 2140 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:42.978525 2164 raft_consensus.cc:2781] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:42.990959 1950 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:42.993112 2149 raft_consensus.cc:2258] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:42.985873 2018 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:43.048372 2124 raft_consensus.cc:3032] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:43.048838 2124 raft_consensus.cc:729] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:43.049613 2124 consensus_queue.cc:260] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:43.050771 2124 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:43.054929 2125 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:43.055711 2076 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:43.056061 1950 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:43.056391 2149 ts_tablet_manager.cc:1905] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:43.056598 1950 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:43.058540 1950 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:43.059690 1803 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:43.063340 2140 raft_consensus.cc:2726] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:43.065521 2137 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:43.021837 2164 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:43.066335 2164 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:43.020341 2061 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:43.070765 31604 raft_consensus.cc:1261] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:43.071499 31604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 2047) took 1788 ms. Trace:
I20260805 22:24:42.996698 2096 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:43.071844 31604 rpcz_store.cc:276] 0805 22:24:41.282661 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:41.368428 (+ 85767us) service_pool.cc:227] Handling call
0805 22:24:41.368694 (+ 266us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:42.965067 (+1596373us) spinlock_profiling.cc:243] Waited 1.01 s on lock 0x7b580046dcb0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2730fe3f63 00007f2731047aba 00007f273105f3af 00007f2731060b4e 00007f273106ac89 00007f273106b556 00007f2731067edf 00007f273106799e 00007f2735fa92e9 00007f272f68f684
0805 22:24:43.071244 (+106177us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:43.071447 (+ 203us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2318608768}
I20260805 22:24:43.083828 31706 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:43.111030 2018 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:43.112959 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:43.113672 31714 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:43.022125 2032 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:43.100137 2160 log.cc:924] Time spent T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.230s user 0.001s sys 0.000s
W20260805 22:24:43.082630 2058 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260805 22:24:43.071756 1888 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:43.121870 2175 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:43.128444 1975 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:43.080983 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:43.209239 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:43.209915 31705 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:43.210916 31713 raft_consensus.cc:2379] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:43.204955 2075 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:43.212504 2030 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:43.212862 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:43.213078 31554 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:43.214336 1780 raft_consensus.cc:2781] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:43.214771 1780 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:43.215252 1780 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:43.122673 31714 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:43.050977 1952 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:43.123862 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:43.125653 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:43.081454 2061 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:43.100633 1951 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:43.166718 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 170756us
I20260805 22:24:43.166647 1915 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:43.124660 2176 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:43.128252 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:43.129609 2179 log.cc:924] Time spent T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.082s user 0.000s sys 0.001s
I20260805 22:24:43.173626 2161 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:43.174021 1886 raft_consensus.cc:3037] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:43.133931 2014 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:43.178814 2096 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:43.181366 31500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40740 (request call id 1075) took 1737 ms. Trace:
I20260805 22:24:43.254280 31500 rpcz_store.cc:276] 0805 22:24:41.443935 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:41.978049 (+534114us) service_pool.cc:227] Handling call
0805 22:24:41.978317 (+ 268us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:42.124531 (+146214us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:42.124564 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:42.124607 (+ 43us) raft_consensus.cc:1631] Marking committed up to 2
0805 22:24:42.124642 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:43.181154 (+1056512us) spinlock_profiling.cc:243] Waited 146 ms on lock 0x7b54002c1fd4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:43.181194 (+ 40us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:43.181310 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":335799552}
W20260805 22:24:43.186108 31711 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1665) took 1685 ms (client timeout 1863 ms). Trace:
W20260805 22:24:43.255522 31711 rpcz_store.cc:269] 0805 22:24:41.500729 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:41.511814 (+ 11085us) service_pool.cc:227] Handling call
0805 22:24:43.185914 (+1674100us) spinlock_profiling.cc:243] Waited 541 ms on lock 0x7b5400314054. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:43.186042 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1243900672}
I20260805 22:24:43.188016 2032 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:43.190310 2138 log.cc:924] Time spent T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.598s user 0.002s sys 0.000s
I20260805 22:24:43.192353 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:43.192792 2085 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:43.196249 31594 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:43.195699 2133 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:43.079588 1958 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:43.123183 1803 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:43.197981 1902 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:43.167389 2018 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:43.336807 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:43.204774 31707 raft_consensus.cc:1261] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:43.135249 2064 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:43.208309 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:43.208438 2078 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:43.308712 31597 raft_consensus.cc:2379] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 4: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:43.220360 2175 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:43.250535 1989 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:43.249265 2130 consensus_queue.cc:1046] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20260805 22:24:43.265311 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:43.265583 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:43.381322 31604 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:43.213284 2030 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:43.303259 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 417215us
I20260805 22:24:43.307801 31705 raft_consensus.cc:1226] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260805 22:24:43.308410 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:43.384261 31667 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:43.262656 2014 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:43.257011 2075 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:43.221900 31554 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:43.360677 31609 raft_consensus.cc:1226] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260805 22:24:43.275022 1975 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:43.249779 1915 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:43.248878 2176 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:43.375186 31601 raft_consensus.cc:2379] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 6: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:43.273458 2125 consensus_queue.cc:1035] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.104s
I20260805 22:24:43.347277 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:43.256505 2164 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:43.375810 31486 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:43.418558 31596 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:43.402774 2130 consensus_queue.cc:1225] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer 516cb07ce3064ccbbca35e912d62816c log is divergent from this leader: its last log entry 3.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
W20260805 22:24:43.400108 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 178286us
W20260805 22:24:43.360019 1838 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.269s user 0.013s sys 0.023s
I20260805 22:24:43.358552 2018 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:43.403934 1949 raft_consensus.cc:2781] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:43.368469 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:43.440167 31444 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:43.445005 31499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1621) took 1218 ms. Trace:
I20260805 22:24:43.454360 31604 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:43.451005 31499 rpcz_store.cc:276] 0805 22:24:42.226745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:42.226864 (+ 119us) service_pool.cc:227] Handling call
0805 22:24:42.227168 (+ 304us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:42.233873 (+ 6705us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:42.233905 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:42.243723 (+ 9818us) log.cc:842] Serialized 26 byte log entry
0805 22:24:42.434491 (+190768us) raft_consensus.cc:1631] Marking committed up to 1
0805 22:24:42.434529 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:42.434603 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:43.189494 (+754891us) spinlock_profiling.cc:243] Waited 3.77 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:24:43.411105 (+221611us) raft_consensus.cc:1671] finished
0805 22:24:43.411126 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:43.444937 (+ 33811us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":3663,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":63568,"spinlock_wait_cycles":8680704,"thread_start_us":128759,"threads_started":1,"wal-append.queue_time_us":223637}
I20260805 22:24:43.456727 32581 raft_consensus.cc:2454] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:43.455456 31444 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:43.457156 1949 raft_consensus.cc:686] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:43.457607 1998 raft_consensus.cc:2781] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:43.457994 1998 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:43.458323 1998 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:43.458110 1949 consensus_queue.cc:237] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:43.435658 1952 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:43.462471 1981 raft_consensus.cc:2781] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:43.463006 1981 raft_consensus.cc:686] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:43.463588 2018 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:43.466063 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 81887.2us
I20260805 22:24:43.467252 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:43.467342 31667 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:24:43.445456 1838 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.357s user 0.013s sys 0.023s
I20260805 22:24:43.469997 32587 raft_consensus.cc:2354] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4.
I20260805 22:24:43.478843 31556 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:43.485780 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:43.486652 31711 raft_consensus.cc:2354] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4.
I20260805 22:24:43.487629 31492 raft_consensus.cc:1261] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:43.487769 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:43.488399 31501 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:43.491314 31501 raft_consensus.cc:1261] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:24:43.492424 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:43.495867 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 55748.8us
I20260805 22:24:43.500610 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:43.501338 31501 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:43.480142 1951 raft_consensus.cc:2726] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:43.501686 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:43.507081 31704 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:43.505162 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:43.507877 31492 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:43.508628 31604 raft_consensus.cc:1226] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5. Ops: [4.4-4.4]
I20260805 22:24:43.505597 31556 leader_election.cc:398] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Vote denied by peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801) with higher term. Message: Invalid argument: T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4.
I20260805 22:24:43.481098 1838 raft_consensus.cc:686] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:43.509172 31556 leader_election.cc:401] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260805 22:24:43.511808 31501 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
I20260805 22:24:43.492548 2031 raft_consensus.cc:2781] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:43.463825 1981 consensus_queue.cc:237] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:43.492449 1754 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:43.495563 2137 raft_consensus.cc:2781] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:43.522418 32529 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:43.517757 31604 raft_consensus.cc:1261] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:24:43.518821 31554 leader_election.cc:398] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4.
I20260805 22:24:43.504034 2058 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:43.552258 2058 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:43.554451 2058 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:43.553966 1754 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:43.555099 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:43.556386 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:43.553838 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:43.512795 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:43.512861 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:43.558075 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:43.558615 31594 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:43.558933 1838 consensus_queue.cc:237] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:43.519655 31444 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:43.562388 31444 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:43.521842 2076 consensus_queue.cc:1035] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.002s
I20260805 22:24:43.522094 2031 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:43.563961 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:43.505115 1902 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:43.510056 2161 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:43.510684 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:43.533502 2187 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:43.495253 2176 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:43.533571 2030 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:43.533767 2133 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:43.534145 1958 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:43.538586 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:43.539739 31604 raft_consensus.cc:1226] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:43.542915 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:43.581241 2010 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:43.587342 31501 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:43.534966 1915 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:43.543754 1931 consensus_queue.cc:1035] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:24:43.544277 1989 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:43.513484 2018 consensus_queue.cc:1035] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.031s
I20260805 22:24:43.546068 2177 raft_consensus.cc:2781] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:24:43.547274 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:43.547978 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:43.549458 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:43.550021 2164 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:43.556973 31713 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:43.599871 31711 raft_consensus.cc:2396] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260805 22:24:43.564328 2031 raft_consensus.cc:3037] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:43.567190 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:43.530232 2175 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:43.565126 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:43.543670 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:43.532604 2064 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:43.570334 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:43.571729 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:43.565806 1982 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:43.578522 31444 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:43.614874 31500 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:43.577665 2076 raft_consensus.cc:2726] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:43.579710 31447 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:43.580428 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:43.588042 2137 raft_consensus.cc:686] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:43.593283 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:43.618618 31604 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:43.619397 31444 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:43.619940 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:43.620597 31604 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:43.573602 1998 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:43.580468 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:43.615969 31494 raft_consensus.cc:2379] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:43.620533 31444 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:43.627710 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:43.628228 31494 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:43.556310 1754 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:43.588845 2149 ts_tablet_manager.cc:1918] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20260805 22:24:43.624783 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:43.630730 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:43.631667 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:43.617038 31493 raft_consensus.cc:2396] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 6 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260805 22:24:43.606940 31667 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:43.622395 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:43.624415 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:43.588718 1905 consensus_queue.cc:1035] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.031s
I20260805 22:24:43.587600 2010 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:43.636796 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:43.610107 2161 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:43.618862 31554 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:43.634363 31500 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:43.635396 31667 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:43.641284 31490 raft_consensus.cc:2379] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 4: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:43.642278 31554 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:43.643356 31554 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:43.643733 31667 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:43.639201 31704 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:43.614794 2177 raft_consensus.cc:686] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:43.615131 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:43.652038 31667 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:43.652635 31554 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:43.655063 31494 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:24:43.656430 31667 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:43.626462 1803 raft_consensus.cc:2726] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:43.637838 1982 raft_consensus.cc:2726] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:43.626191 2085 raft_consensus.cc:2726] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:43.669606 31667 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:43.629504 2064 raft_consensus.cc:2781] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:43.615564 1919 consensus_queue.cc:1035] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.038s
I20260805 22:24:43.655167 2149 log.cc:1196] T e6cf90ae3fc840ceb48be8c790be97d9 P 516cb07ce3064ccbbca35e912d62816c: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-1-root/wals/e6cf90ae3fc840ceb48be8c790be97d9
I20260805 22:24:43.641997 1754 raft_consensus.cc:2726] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:43.646430 2078 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:43.646210 2164 raft_consensus.cc:2726] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:43.752784 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:43.651647 2080 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:43.659211 1950 raft_consensus.cc:2781] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:43.654748 1952 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:43.659591 1958 raft_consensus.cc:2726] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260805 22:24:43.733958 31501 raft_consensus.cc:1226] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4. Ops: [3.2-3.2]
I20260805 22:24:43.728899 2124 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:43.641584 1780 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:43.647480 2125 consensus_queue.cc:1035] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.108s
I20260805 22:24:43.729130 2193 raft_consensus.cc:2726] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:43.654954 2075 consensus_queue.cc:1035] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.104s
I20260805 22:24:43.738102 2096 raft_consensus.cc:2726] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:43.647414 2161 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:43.877924 31594 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:24:43.750856 2064 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:43.887498 2149 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59024 (request call id 1080) took 1365 ms. Trace:
I20260805 22:24:43.880554 1950 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:43.882794 1952 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:43.898605 2010 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:43.872680 2078 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260805 22:24:43.911441 2214 log.cc:924] Time spent T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.256s user 0.001s sys 0.000s
I20260805 22:24:43.884958 2124 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260805 22:24:43.940331 1886 consensus_meta.cc:287] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.684s user 0.011s sys 0.000s
I20260805 22:24:43.932725 2064 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:43.992396 31602 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
W20260805 22:24:44.003170 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 359226us
I20260805 22:24:43.953544 1950 raft_consensus.cc:3037] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:44.004004 31444 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:43.952488 2078 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:43.799280 2137 consensus_queue.cc:237] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:43.954464 1952 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:44.018425 31444 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:43.885152 2080 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:43.978302 2124 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:43.993870 2176 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:43.902252 2177 consensus_queue.cc:237] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:44.154129 2080 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:24:44.155349 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 136995us
W20260805 22:24:44.155634 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 486064us
I20260805 22:24:44.157048 31444 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:44.157400 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:44.158376 31594 raft_consensus.cc:2379] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 6: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:44.088199 2078 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:44.081403 1905 raft_consensus.cc:2781] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:44.161530 2176 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:44.121181 1998 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:44.149636 2076 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:44.127293 2125 raft_consensus.cc:2726] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260805 22:24:44.179942 31492 consensus_meta.cc:287] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.672s user 0.011s sys 0.004s
I20260805 22:24:44.179549 31667 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:44.184533 31444 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:44.186636 31492 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:24:44.192298 1780 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:44.185259 1958 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:44.176332 1905 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:44.207340 31667 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:44.200567 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:44.208323 31594 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:44.190464 2085 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:44.228489 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:44.184902 1951 consensus_queue.cc:1046] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
W20260805 22:24:44.229436 2210 log.cc:924] Time spent T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.345s user 0.000s sys 0.001s
I20260805 22:24:44.193859 1998 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:44.230206 1998 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:44.230931 2218 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:44.231498 2218 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:44.232372 1998 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:44.234212 2213 log.cc:924] Time spent T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.255s user 0.000s sys 0.001s
I20260805 22:24:44.193255 1958 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:44.229341 31704 raft_consensus.cc:2379] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:44.241268 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:44.242098 32591 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:43.933418 2149 rpcz_store.cc:276] 0805 22:24:42.544018 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:42.544263 (+ 245us) service_pool.cc:227] Handling call
0805 22:24:42.760757 (+216494us) ts_tablet_manager.cc:1145] Deleting tablet e6cf90ae3fc840ceb48be8c790be97d9
0805 22:24:42.801413 (+ 40656us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0805 22:24:43.437134 (+635721us) tablet_metadata.cc:641] Metadata flushed
0805 22:24:43.565989 (+128855us) tablet_metadata.cc:641] Metadata flushed
0805 22:24:43.887415 (+321426us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":15754,"tablet-delete.queue_time_us":148393,"thread_start_us":86952,"threads_started":1}
W20260805 22:24:44.228838 2211 log.cc:924] Time spent T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.346s user 0.001s sys 0.000s
I20260805 22:24:44.210172 2096 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:44.206713 1780 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:44.209620 1996 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:44.210680 1905 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:44.189616 2076 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:44.244980 2207 log.cc:924] Time spent T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.366s user 0.001s sys 0.001s
I20260805 22:24:44.245982 2158 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:44.226210 2085 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:44.258684 31339 catalog_manager.cc:4841] TS 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): tablet e6cf90ae3fc840ceb48be8c790be97d9 (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:44.252197 2218 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:44.254536 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:44.265978 32581 raft_consensus.cc:2379] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 4: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:44.270787 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:44.256116 2161 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:44.258019 1996 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:44.274251 31594 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:44.277204 2078 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.279227 2078 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:24:44.282235 2192 log.cc:924] Time spent T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.757s user 0.005s sys 0.009s
I20260805 22:24:44.285552 31444 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:44.285737 31447 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:44.286585 1982 raft_consensus.cc:2781] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:44.287104 1982 raft_consensus.cc:686] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:44.288041 1982 consensus_queue.cc:237] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.290022 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:44.291661 2076 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:44.258137 2222 raft_consensus.cc:2781] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:44.295463 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:44.263258 2158 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:44.298992 31444 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:44.270215 2085 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:44.259932 1780 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.309840 1998 raft_consensus.cc:2726] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:44.312811 31714 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58348 (request call id 398) took 1055 ms. Trace:
I20260805 22:24:44.313211 31714 rpcz_store.cc:276] 0805 22:24:43.257758 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:43.257901 (+ 143us) service_pool.cc:227] Handling call
0805 22:24:43.258231 (+ 330us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:43.263974 (+ 5743us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:43.264002 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:43.268973 (+ 4971us) log.cc:842] Serialized 26 byte log entry
0805 22:24:43.586766 (+317793us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:24:43.586801 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:43.586863 (+ 62us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:44.312571 (+725708us) raft_consensus.cc:1671] finished
0805 22:24:44.312595 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:44.312745 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":203669,"raft.queue_time_us":5873,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":35135,"thread_start_us":218142,"threads_started":1,"wal-append.queue_time_us":323062}
I20260805 22:24:44.310225 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:44.257745 2096 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:44.272037 1958 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:44.296448 2076 raft_consensus.cc:2781] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:44.295141 1996 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:44.293617 2022 raft_consensus.cc:3032] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:44.297411 2222 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:44.337061 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:24:44.341460 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 134196us
I20260805 22:24:44.342296 31667 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:44.300964 2158 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:44.343822 31667 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:44.345921 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:44.347105 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:44.347918 31704 raft_consensus.cc:2379] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 4: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:44.327207 2161 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:44.355521 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:44.356457 31492 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 6
I20260805 22:24:44.356900 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:44.350023 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:44.357653 31708 raft_consensus.cc:2379] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 4: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:44.358472 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:44.349367 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:44.350133 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:44.356094 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:44.359745 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:44.360297 31714 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:44.360822 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:44.350780 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:44.362057 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:44.362715 31711 raft_consensus.cc:2354] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6.
I20260805 22:24:44.363147 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:44.363363 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:44.363864 31711 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 5
I20260805 22:24:44.364454 31444 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:44.365633 1803 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:44.367115 31492 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:24:44.340955 2022 raft_consensus.cc:729] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:44.347805 2078 raft_consensus.cc:2726] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:44.368636 31502 raft_consensus.cc:1226] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:44.369300 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:44.369452 32586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57372 (request call id 560) took 1098 ms. Trace:
I20260805 22:24:44.341668 2076 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:44.373980 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:44.342298 2222 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:44.374692 2161 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.393702 2161 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:44.394210 2161 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:44.394673 2161 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:44.396677 2161 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:44.376967 2187 consensus_meta.cc:287] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.811s user 0.010s sys 0.000s
I20260805 22:24:44.397311 2187 raft_consensus.cc:2726] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801) with higher term. Message: Invalid argument: T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4.
I20260805 22:24:44.377935 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:44.399173 31707 raft_consensus.cc:1226] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260805 22:24:44.400957 2076 raft_consensus.cc:3037] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:44.401288 32581 raft_consensus.cc:1261] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260805 22:24:44.379019 31444 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:44.401928 1780 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:44.402416 2227 raft_consensus.cc:2726] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:44.401559 31601 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:44.402441 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:44.386278 31447 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:44.399236 31713 raft_consensus.cc:2354] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5.
I20260805 22:24:44.380493 31594 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:44.381312 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:44.425274 31711 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:44.428627 31494 raft_consensus.cc:1261] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:24:44.429519 31711 raft_consensus.cc:3032] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:44.430029 31711 raft_consensus.cc:729] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:44.431537 31594 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:24:44.432688 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:44.437819 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:44.442432 1950 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:44.443648 2218 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:44.444008 31715 raft_consensus.cc:1226] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:44.385119 32586 rpcz_store.cc:276] 0805 22:24:43.270794 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:43.272378 (+ 1584us) service_pool.cc:227] Handling call
0805 22:24:44.368687 (+1096309us) spinlock_profiling.cc:243] Waited 11.8 ms on lock 0x7b5800005198. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108b7c 00007f272e10873b 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:24:44.368868 (+ 181us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:44.369075 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:44.369101 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:44.369146 (+ 45us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:24:44.369193 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:44.369248 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:44.369354 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27042560}
W20260805 22:24:44.443003 31602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56384 (request call id 288) took 1733 ms (client timeout 2020 ms). Trace:
W20260805 22:24:44.445586 31602 rpcz_store.cc:269] 0805 22:24:42.709479 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:42.709855 (+ 376us) service_pool.cc:227] Handling call
0805 22:24:43.212704 (+502849us) spinlock_profiling.cc:243] Waited 8.83 ms on lock 0x7b5800004598. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735faa1b0 00007f2735fa9a94 00007f272f691c04 00007f272f691b92
0805 22:24:44.442846 (+1230142us) spinlock_profiling.cc:243] Waited 779 ms on lock 0x7b540026fcd4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:44.442947 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1811331200}
I20260805 22:24:44.447860 31714 raft_consensus.cc:2396] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for term 5 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260805 22:24:44.453169 31714 raft_consensus.cc:1261] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:24:44.385489 1951 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:24:44.380837 2137 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:44.430855 31711 consensus_queue.cc:260] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.460340 31711 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:44.468096 1951 raft_consensus.cc:686] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:44.470026 31667 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:44.470503 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:44.392694 2022 consensus_queue.cc:260] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:44.490736 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:44.492564 31594 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:44.494236 31667 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:44.496294 31711 raft_consensus.cc:1261] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 2. (term mismatch)
I20260805 22:24:44.496804 31711 pending_rounds.cc:76] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385: Aborting all ops after (but not including) 1
I20260805 22:24:44.497195 31711 pending_rounds.cc:97] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385: Aborting uncommitted NO_OP operation due to leader change: 3.2
I20260805 22:24:44.497231 2161 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:44.497644 31711 raft_consensus.cc:2866] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:24:44.504287 31667 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:44.505915 31594 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:24:44.508471 31594 raft_consensus.cc:1261] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 2. (index mismatch)
I20260805 22:24:44.445863 2078 consensus_queue.cc:1035] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260805 22:24:44.513447 31667 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:44.519119 2227 consensus_queue.cc:1035] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:24:44.503096 2228 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:44.469014 1951 consensus_queue.cc:237] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.510677 2022 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:44.507885 2234 raft_consensus.cc:2726] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260805 22:24:44.525738 2064 consensus_meta.cc:287] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.520s user 0.009s sys 0.000s
I20260805 22:24:44.469363 2137 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:44.496590 1905 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:44.546298 31603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 2072) took 1107 ms. Trace:
I20260805 22:24:44.523654 2244 raft_consensus.cc:2726] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:44.549661 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:44.534102 2161 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:24:44.547698 31603 rpcz_store.cc:276] 0805 22:24:43.438719 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:43.439964 (+ 1245us) service_pool.cc:227] Handling call
0805 22:24:43.440309 (+ 345us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:43.946848 (+506539us) spinlock_profiling.cc:243] Waited 1.22 ms on lock 0x7b54001d63a0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2730ff8730 00007f273106a300 00007f273106b371 00007f2731067edf 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc
0805 22:24:43.961811 (+ 14963us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:43.961842 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:43.962341 (+ 499us) log.cc:842] Serialized 26 byte log entry
0805 22:24:44.347554 (+385213us) raft_consensus.cc:1631] Marking committed up to 2
0805 22:24:44.347688 (+ 134us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:44.347761 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:44.546023 (+198262us) raft_consensus.cc:1671] finished
0805 22:24:44.546042 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:44.546239 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":15484,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":11772,"spinlock_wait_cycles":2812672,"thread_start_us":364794,"threads_started":1,"wal-append.queue_time_us":424940}
I20260805 22:24:44.526250 2133 consensus_queue.cc:1035] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260805 22:24:44.522497 1952 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.562315 1952 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:44.562388 31600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56384 (request call id 292) took 1065 ms. Trace:
I20260805 22:24:44.559741 2061 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:44.549198 2096 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:44.560520 2014 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:44.562839 31600 rpcz_store.cc:276] 0805 22:24:43.496595 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:44.561481 (+1064886us) service_pool.cc:227] Handling call
0805 22:24:44.561865 (+ 384us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:44.562044 (+ 179us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:44.562079 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:44.562123 (+ 44us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:24:44.562164 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:44.562219 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:44.562327 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:24:44.564033 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:44.555543 1998 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:44.566771 1998 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.568701 1998 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:44.570581 2137 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:44.572510 1905 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:44.575950 2031 consensus_meta.cc:287] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.958s user 0.006s sys 0.006s
I20260805 22:24:44.576406 2031 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:44.576997 31500 raft_consensus.cc:1226] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:44.564779 2158 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.579908 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:44.580669 31596 raft_consensus.cc:2354] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4.
I20260805 22:24:44.584046 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:44.584656 32587 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:44.585747 31668 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:24:44.587826 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 67890.4us
I20260805 22:24:44.549081 1950 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:44.527779 2228 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.594813 31704 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:44.578121 2161 raft_consensus.cc:686] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:44.547566 2064 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:44.600013 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 66303.5us
I20260805 22:24:44.592365 2078 raft_consensus.cc:2781] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:44.609776 2061 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:44.610004 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:44.578277 2124 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.612663 31667 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:44.597834 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:44.615564 31444 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:44.610364 2061 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.618670 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:44.625739 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:44.626214 2244 raft_consensus.cc:2781] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:24:44.626806 2244 raft_consensus.cc:686] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:44.627753 2244 consensus_queue.cc:237] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.629114 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:44.630752 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:44.631453 31502 raft_consensus.cc:2379] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 3: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:44.633198 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:44.633824 31707 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:44.633673 31667 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:44.636580 2064 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:44.627034 2096 consensus_queue.cc:1046] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20260805 22:24:44.639992 2124 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:44.592918 2076 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:44.643510 2076 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:44.643761 2227 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:44.644085 2076 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:44.644373 2227 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:44.645788 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:44.646320 1982 raft_consensus.cc:2726] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:44.646569 2158 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:44.647369 2227 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:44.648739 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:44.649339 31502 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:44.649732 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:44.650321 32587 raft_consensus.cc:3037] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 5
I20260805 22:24:44.652438 2137 raft_consensus.cc:2726] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260805 22:24:44.653249 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:44.635874 31667 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:44.644605 2076 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:44.661887 2201 log.cc:924] Time spent T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 1.137s user 0.001s sys 0.004s
I20260805 22:24:44.663169 31608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 2062) took 1981 ms. Trace:
I20260805 22:24:44.663524 31608 rpcz_store.cc:276] 0805 22:24:42.681997 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:42.799685 (+117688us) service_pool.cc:227] Handling call
0805 22:24:43.163838 (+364153us) spinlock_profiling.cc:243] Waited 13.7 ms on lock 0x7b5800004598. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:24:43.164052 (+ 214us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:43.214717 (+ 50665us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:43.214746 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:43.215279 (+ 533us) log.cc:842] Serialized 26 byte log entry
0805 22:24:43.466825 (+251546us) raft_consensus.cc:1631] Marking committed up to 1
0805 22:24:43.466865 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:43.466938 (+ 73us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:44.661053 (+1194115us) spinlock_profiling.cc:243] Waited 11.4 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:24:44.662940 (+ 1887us) raft_consensus.cc:1671] finished
0805 22:24:44.662962 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:44.663105 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":50237,"raft.queue_time_us":50592,"raft.run_cpu_time_us":311,"raft.run_wall_time_us":352,"spinlock_wait_cycles":57754880,"thread_start_us":251087,"threads_started":1,"wal-append.queue_time_us":296998}
I20260805 22:24:44.666038 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:44.666720 31608 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:44.668202 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:44.611326 2078 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:44.674863 31444 leader_election.cc:398] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6.
I20260805 22:24:44.675210 31444 leader_election.cc:401] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Cancelling election due to peer responding with higher term
I20260805 22:24:44.676975 31444 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:44.678175 31444 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:44.674188 2130 consensus_queue.cc:1035] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
I20260805 22:24:44.615126 31486 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40740 (request call id 1080) took 1101 ms. Trace:
I20260805 22:24:44.682269 31444 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:44.683552 31444 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:44.684629 31500 raft_consensus.cc:1226] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:44.681030 1920 raft_consensus.cc:2726] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6.
I20260805 22:24:44.692924 2217 consensus_queue.cc:1046] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:24:44.694185 2217 raft_consensus.cc:3032] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:24:44.668917 2076 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:44.686086 2227 raft_consensus.cc:2726] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:44.695698 2064 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:44.697290 2254 raft_consensus.cc:2726] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:44.698130 1982 raft_consensus.cc:2726] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:44.698577 31444 leader_election.cc:398] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5.
I20260805 22:24:44.698858 31444 leader_election.cc:401] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Cancelling election due to peer responding with higher term
W20260805 22:24:44.697345 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 61544.8us
W20260805 22:24:44.700243 31597 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59028 (request call id 2066) took 1815 ms (client timeout 2188 ms). Trace:
I20260805 22:24:44.701623 2076 raft_consensus.cc:2726] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 4. Reason: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5.
I20260805 22:24:44.701124 31667 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:44.703055 2004 raft_consensus.cc:2726] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:44.705917 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:44.706655 31594 raft_consensus.cc:2354] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5.
I20260805 22:24:44.707836 31444 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:44.711040 2064 raft_consensus.cc:2726] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:44.694787 1998 raft_consensus.cc:2726] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:44.713317 2061 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:44.714943 2228 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:24:44.630074 2031 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:44.716557 31667 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:44.720695 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:44.722307 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:44.723001 31608 raft_consensus.cc:3037] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:44.726365 31667 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:44.673308 2078 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:44.736778 31667 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:44.630364 1996 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.694669 2217 raft_consensus.cc:729] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:44.752540 31444 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:44.750808 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:44.700771 31597 rpcz_store.cc:269] 0805 22:24:42.885001 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:43.011642 (+126641us) service_pool.cc:227] Handling call
0805 22:24:44.699877 (+1688235us) spinlock_profiling.cc:243] Waited 40.3 ms on lock 0x7b540005d2d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:44.700154 (+ 277us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":92692736}
I20260805 22:24:44.759120 31494 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:44.763231 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:44.737653 2254 raft_consensus.cc:2726] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:44.759915 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:44.766499 31707 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:44.721767 2228 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:44.724080 2064 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:44.751082 1950 raft_consensus.cc:2781] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:44.769003 32587 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:44.689843 31486 rpcz_store.cc:276] 0805 22:24:43.512854 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:44.608558 (+1095704us) service_pool.cc:227] Handling call
0805 22:24:44.613926 (+ 5368us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:24:44.626798 2161 consensus_queue.cc:237] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.772400 31502 raft_consensus.cc:2354] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5.
I20260805 22:24:44.691797 2085 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.777062 31447 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:44.778046 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:44.717063 2224 consensus_queue.cc:1046] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.044s
I20260805 22:24:44.735903 2076 consensus_queue.cc:1035] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.030s
I20260805 22:24:44.761166 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:44.763593 2218 raft_consensus.cc:2726] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:44.767756 2193 raft_consensus.cc:2726] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:44.755059 1919 consensus_queue.cc:1046] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260805 22:24:44.775555 1950 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:44.806813 31444 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:44.807933 31708 raft_consensus.cc:1226] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:44.783349 1989 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:44.780813 2064 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.835800 1989 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:44.837821 2064 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:44.838671 1989 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:44.846923 31486 raft_consensus.cc:1261] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260805 22:24:44.849550 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:44.850482 31492 raft_consensus.cc:2379] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:44.852880 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:44.853752 31494 raft_consensus.cc:2379] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:44.855647 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:44.856283 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:44.865283 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:44.866127 31492 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:44.867847 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:44.869045 31499 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:44.827101 2222 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.809547 1950 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:44.843253 1996 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:44.867830 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:44.780267 2228 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.772056 2252 consensus_queue.cc:1046] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.086s
W20260805 22:24:44.870771 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 103076us
I20260805 22:24:44.811316 2227 raft_consensus.cc:2781] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:44.874821 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:44.874080 31494 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:44.876973 31499 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:44.843106 2176 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:44.872115 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:44.885356 31707 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:44.886955 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:44.879311 31499 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1641) took 1392 ms (client timeout 1819 ms). Trace:
I20260805 22:24:44.881220 31554 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:44.889004 31499 rpcz_store.cc:269] 0805 22:24:43.487181 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:44.867550 (+1380369us) service_pool.cc:227] Handling call
0805 22:24:44.879259 (+ 11709us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23040}
I20260805 22:24:44.890508 31554 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:44.890793 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:44.892756 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:44.893239 31499 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:44.849000 2085 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:44.896581 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:44.871307 2224 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:44.796866 2217 consensus_queue.cc:260] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:44.897727 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:44.929877 31709 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260805 22:24:45.030165 31707 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
W20260805 22:24:44.936746 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 44123.4us
I20260805 22:24:44.902292 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:44.898099 32587 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260805 22:24:45.032294 31608 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:45.031278 31554 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:45.033607 31714 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:45.033759 31554 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:44.899075 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 92362us
I20260805 22:24:45.034833 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:45.035696 32591 raft_consensus.cc:2379] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:45.036401 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:45.047077 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:45.037894 31667 leader_election.cc:398] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5.
I20260805 22:24:45.051935 31667 leader_election.cc:401] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260805 22:24:45.038486 31494 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:45.051888 31714 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:45.042107 31554 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:45.045742 2124 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:45.054662 1952 raft_consensus.cc:2726] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:24:45.027552 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:44.926323 2125 raft_consensus.cc:2726] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:45.056020 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:45.056396 32584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57372 (request call id 567) took 1518 ms. Trace:
I20260805 22:24:45.058015 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:45.059052 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:45.059268 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:44.894621 2052 raft_consensus.cc:2726] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:45.045763 2030 raft_consensus.cc:2781] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:45.064368 2030 raft_consensus.cc:686] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:45.065269 2030 consensus_queue.cc:237] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:45.056931 32584 rpcz_store.cc:276] 0805 22:24:43.537865 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:43.549129 (+ 11264us) service_pool.cc:227] Handling call
0805 22:24:45.056117 (+1506988us) spinlock_profiling.cc:243] Waited 738 ms on lock 0x7b54004303d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:45.056312 (+ 195us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1698006784}
I20260805 22:24:44.898662 2031 raft_consensus.cc:2726] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:45.067023 31707 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:45.068413 31554 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:44.984364 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:45.069330 31714 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:45.070884 31486 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:24:45.071170 2193 raft_consensus.cc:2726] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5.
I20260805 22:24:44.959760 2227 raft_consensus.cc:686] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:45.071655 2181 raft_consensus.cc:3032] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:45.072048 2181 raft_consensus.cc:729] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:45.072439 2227 consensus_queue.cc:237] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.072858 2181 consensus_queue.cc:260] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.070099 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:45.074024 31704 raft_consensus.cc:2379] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:45.078879 31501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40740 (request call id 1081) took 1457 ms. Trace:
I20260805 22:24:45.079260 31501 rpcz_store.cc:276] 0805 22:24:43.620999 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:43.621694 (+ 695us) service_pool.cc:227] Handling call
0805 22:24:43.622016 (+ 322us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:43.883127 (+261111us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:45.078643 (+1195516us) spinlock_profiling.cc:243] Waited 37.2 ms on lock 0x7b5400330754. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:45.078818 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":85501696}
I20260805 22:24:45.083940 1989 consensus_queue.cc:1046] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260805 22:24:45.084200 2158 raft_consensus.cc:3032] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:45.084862 1989 raft_consensus.cc:3032] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:24:45.085242 1989 raft_consensus.cc:729] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:45.086077 1989 consensus_queue.cc:260] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.074095 2058 raft_consensus.cc:2781] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
W20260805 22:24:45.071022 2220 raft_consensus.cc:922] Couldn't advance consensus term.: Illegal state: Can't advance term to: 4 current term: 4 is higher.
I20260805 22:24:45.084794 2158 raft_consensus.cc:729] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:45.038244 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:45.102125 31608 raft_consensus.cc:2379] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:45.092154 31554 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:45.092721 31667 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:45.092737 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:45.104619 31604 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:45.105144 31667 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:45.094971 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 44559.6us
I20260805 22:24:45.112524 32529 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:45.097818 31609 raft_consensus.cc:1261] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:45.053406 31444 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:45.113890 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 60592.7us
I20260805 22:24:45.114509 1920 raft_consensus.cc:2726] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:45.111712 31604 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:45.054616 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:45.015959 2176 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:44.927021 2222 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:45.099141 2058 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:45.140054 1741 raft_consensus.cc:2726] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:45.155560 2031 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:45.210088 31554 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:45.158797 2220 raft_consensus.cc:2781] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:45.159968 2228 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:45.045977 2224 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.214411 31554 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:45.139707 2234 raft_consensus.cc:2726] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:45.162631 2176 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.217006 2030 raft_consensus.cc:2781] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election won for term 6
W20260805 22:24:45.218122 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 113050us
I20260805 22:24:45.214362 31597 raft_consensus.cc:2379] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:45.219444 2181 raft_consensus.cc:3037] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:45.392977 2137 consensus_queue.cc:1035] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:45.222566 2268 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:45.401646 2268 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.222358 1989 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Advancing to term 7
W20260805 22:24:45.225082 2195 log.cc:924] Time spent T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0: Append to log took a long time: real 1.699s user 0.016s sys 0.000s
I20260805 22:24:45.213333 2031 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:45.407752 2031 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:45.409034 31609 raft_consensus.cc:1261] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260805 22:24:45.308173 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 93844.3us
I20260805 22:24:45.333190 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:45.161347 2052 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.052s
I20260805 22:24:45.338853 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:45.389623 1902 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:45.390458 2161 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:45.391407 31667 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:45.392002 2222 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:45.392472 2193 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:45.521870 2161 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:45.522315 2193 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:45.523804 2254 raft_consensus.cc:2781] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:45.524452 2193 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:45.524540 2254 raft_consensus.cc:686] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
W20260805 22:24:45.521935 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 130621us
W20260805 22:24:45.527434 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 401002us
I20260805 22:24:45.409977 31608 raft_consensus.cc:1226] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5. Ops: []
W20260805 22:24:45.412907 2280 log.cc:924] Time spent T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.227s user 0.000s sys 0.000s
I20260805 22:24:45.175585 2058 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
W20260805 22:24:45.414939 2211 log.cc:924] Time spent T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.208s user 0.000s sys 0.000s
I20260805 22:24:45.417636 2030 raft_consensus.cc:686] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 6 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
W20260805 22:24:45.421757 1950 consensus_meta.cc:287] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.538s user 0.003s sys 0.006s
W20260805 22:24:45.236986 2263 log.cc:924] Time spent T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260805 22:24:45.160903 2078 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.484937 31711 raft_consensus.cc:2454] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:45.498075 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:45.329295 2228 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:45.242861 2158 consensus_queue.cc:260] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:45.528570 32529 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:45.557894 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:24:45.580490 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 51999.4us
I20260805 22:24:45.557819 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:45.564440 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 78274.5us
I20260805 22:24:45.561940 31609 raft_consensus.cc:1261] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260805 22:24:45.581305 31705 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:45.539743 1902 raft_consensus.cc:686] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:45.559770 2076 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:45.589978 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:45.596391 2267 raft_consensus.cc:2741] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260805 22:24:45.574681 2161 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:45.596959 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:45.606527 2176 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:45.607950 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 155968us
I20260805 22:24:45.560151 2222 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:45.595917 2268 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:45.599335 2032 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:45.582021 1950 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.609601 1981 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:45.592109 2228 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:45.602927 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:45.613762 1949 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:45.633744 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:45.660893 32584 raft_consensus.cc:1799] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Handling vote request from an unknown peer feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:45.604821 2076 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:45.673106 32584 raft_consensus.cc:2396] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260805 22:24:45.676756 31706 raft_consensus.cc:1226] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260805 22:24:45.679204 31714 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:45.635612 1986 raft_consensus.cc:3032] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Stepping down as leader of term 2
W20260805 22:24:45.689921 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 124134us
I20260805 22:24:45.615918 2254 consensus_queue.cc:237] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:45.699764 2078 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:45.699887 1950 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:45.606808 1902 consensus_queue.cc:237] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:45.701344 31444 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:45.634079 2032 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:45.711265 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 169292us
I20260805 22:24:45.700232 2224 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:45.689146 1986 raft_consensus.cc:729] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:45.632385 2030 consensus_queue.cc:237] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:45.653344 1981 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:45.719218 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 126775us
I20260805 22:24:45.722358 31554 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:45.661871 1949 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.724424 31667 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:45.711100 2276 raft_consensus.cc:2781] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:45.717176 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:45.739382 31554 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:45.731665 2193 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:45.741554 31554 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:45.743287 2052 raft_consensus.cc:2726] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:45.726140 2124 consensus_queue.cc:1035] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.018s
I20260805 22:24:45.743906 2052 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:45.744266 2052 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:45.744590 2052 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:45.745656 2276 raft_consensus.cc:686] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:45.745982 2228 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:45.728742 2293 consensus_queue.cc:1046] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
I20260805 22:24:45.746583 2276 consensus_queue.cc:237] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.747481 2293 raft_consensus.cc:3032] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:45.747855 2293 raft_consensus.cc:729] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:45.748633 2293 consensus_queue.cc:260] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:45.749874 32584 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:45.750191 2032 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:45.751915 2052 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.719245 2175 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:45.721818 1958 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:45.759464 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:45.760536 2220 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:45.725831 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:45.761103 1986 consensus_queue.cc:260] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.766119 1986 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:45.767874 2224 raft_consensus.cc:3032] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:45.768245 2224 raft_consensus.cc:729] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:45.755162 2124 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:45.729142 2078 consensus_queue.cc:1035] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260805 22:24:45.769034 2224 consensus_queue.cc:260] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.769690 2124 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.770561 2222 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:45.754583 2296 raft_consensus.cc:2726] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:45.771832 1981 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:45.757277 2228 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:45.764564 2293 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:45.772758 2228 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:45.775216 2228 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:45.777132 1949 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:45.757998 2175 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.779747 2175 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:45.726999 1950 consensus_queue.cc:1035] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.035s
W20260805 22:24:45.786746 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 74713.4us
I20260805 22:24:45.764475 2076 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:45.761487 1958 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.801514 31556 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:45.803063 31556 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:45.812232 31444 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:45.760979 2220 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:45.824918 1981 raft_consensus.cc:2781] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:45.825307 1981 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:45.825680 1981 raft_consensus.cc:3037] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:45.821208 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:45.828269 31492 raft_consensus.cc:2379] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:45.828444 31444 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:45.771744 2124 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:45.832399 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:45.834440 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:45.835155 32594 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
W20260805 22:24:45.837833 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 83329.1us
I20260805 22:24:45.846016 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:45.818608 2061 raft_consensus.cc:2726] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:45.852083 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:45.822539 1949 raft_consensus.cc:2726] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:24:45.858580 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:45.860376 2076 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:45.862114 31556 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:45.863485 2076 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.866688 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:45.837312 1998 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:45.818657 2175 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.043s
I20260805 22:24:45.822206 2031 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:45.878571 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 137082us
I20260805 22:24:45.872802 2061 raft_consensus.cc:2781] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:45.876127 2175 consensus_queue.cc:1225] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer 6bf8c694a0ba40978195ae044a917caf log is divergent from this leader: its last log entry 4.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260805 22:24:45.873663 1998 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:45.874063 2076 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:45.884425 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:45.885017 31608 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:45.890672 2031 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:45.891641 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:45.892298 31604 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:45.892879 2224 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:45.880317 2052 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:45.891963 2058 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.892915 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:45.913859 31594 raft_consensus.cc:1261] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260805 22:24:45.914081 31604 raft_consensus.cc:1226] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260805 22:24:45.913527 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:45.915774 2313 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:45.916913 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:45.916676 2313 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:45.917654 31600 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:45.883198 2061 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:45.921089 2061 raft_consensus.cc:3037] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:45.922719 2058 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:45.886976 1998 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:45.927457 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:45.928442 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:45.915668 1958 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:45.786177 2196 consensus_queue.cc:1159] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "feb5e7ab4b1541beaf2674c3c5500385" responder_term: 3 status { last_received { term: 2 index: 5 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260805 22:24:45.937108 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:45.987658 32586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57372 (request call id 571) took 1105 ms. Trace:
I20260805 22:24:45.973253 2220 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:45.990183 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:45.991004 31602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59028 (request call id 2090) took 1232 ms. Trace:
I20260805 22:24:45.945724 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:45.991335 31602 rpcz_store.cc:276] 0805 22:24:44.758711 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:44.758854 (+ 143us) service_pool.cc:227] Handling call
0805 22:24:45.989323 (+1230469us) spinlock_profiling.cc:243] Waited 3.54 ms on lock 0x7b5800004598. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735faa1b0 00007f2735fa9a94 00007f272f691c04 00007f272f691b92
0805 22:24:45.990941 (+ 1618us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8135424}
I20260805 22:24:45.988098 32586 rpcz_store.cc:276] 0805 22:24:44.882107 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:44.884289 (+ 2182us) service_pool.cc:227] Handling call
0805 22:24:45.986793 (+1102504us) spinlock_profiling.cc:243] Waited 6.74 ms on lock 0x7b5800005198. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:24:45.987032 (+ 239us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:45.987259 (+ 227us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:45.987291 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:45.987354 (+ 63us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:24:45.987391 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:45.987452 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:45.987594 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":15508736}
I20260805 22:24:45.997720 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:45.998353 31492 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:45.999374 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:45.999836 2061 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:46.000378 31601 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59030 (request call id 1069) took 2443 ms (client timeout 2695 ms). Trace:
W20260805 22:24:46.000768 31601 rpcz_store.cc:269] 0805 22:24:43.556634 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:43.566932 (+ 10298us) service_pool.cc:227] Handling call
0805 22:24:46.000194 (+2433262us) spinlock_profiling.cc:243] Waited 791 ms on lock 0x7b5400050ad4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:46.000321 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1819644800}
I20260805 22:24:46.004084 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:46.005105 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:45.940551 31608 raft_consensus.cc:1226] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260805 22:24:46.005863 31704 raft_consensus.cc:2379] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:46.009934 31603 raft_consensus.cc:1226] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:46.010913 31444 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:45.973122 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:24:46.016671 31596 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59030 (request call id 1072) took 2459 ms (client timeout 2243 ms). Trace:
W20260805 22:24:46.017148 31596 rpcz_store.cc:269] 0805 22:24:43.557727 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:43.564896 (+ 7169us) service_pool.cc:227] Handling call
0805 22:24:46.004144 (+2439248us) spinlock_profiling.cc:243] Waited 963 ms on lock 0x7b54002373d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:46.016587 (+ 12443us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2215936256}
I20260805 22:24:45.934113 2269 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:45.935348 2309 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:45.971789 2313 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:46.038028 31495 raft_consensus.cc:1226] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:45.949949 32581 pending_rounds.cc:76] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf: Aborting all ops after (but not including) 5
I20260805 22:24:46.023115 2064 raft_consensus.cc:2726] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:46.046456 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:46.044406 2309 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:46.047274 32586 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:60186
I20260805 22:24:46.044452 32581 pending_rounds.cc:97] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260805 22:24:46.050877 32581 raft_consensus.cc:2866] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:24:46.053314 31596 raft_consensus.cc:2354] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4.
I20260805 22:24:46.049187 2309 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:46.055441 1981 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:46.060102 32584 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:46.061770 31668 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:46.046933 31495 raft_consensus.cc:1226] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:46.031780 2175 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:46.079780 2175 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:46.035885 2220 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:24:46.036556 2269 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:46.088145 2269 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:46.088827 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:46.096254 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:46.101264 1981 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:46.104344 31500 raft_consensus.cc:1261] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260805 22:24:46.104773 31608 raft_consensus.cc:1261] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260805 22:24:46.105505 31444 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:46.106433 31601 raft_consensus.cc:1261] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:24:46.106725 1981 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:46.107268 1981 raft_consensus.cc:686] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:46.108151 1981 consensus_queue.cc:237] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:46.074518 2164 raft_consensus.cc:2781] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:46.106669 2015 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:46.117820 2164 raft_consensus.cc:686] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:46.117971 2015 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:46.118777 2164 consensus_queue.cc:237] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:46.076053 2217 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:46.078296 32586 raft_consensus.cc:595] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:46.109465 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:46.110209 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:46.124089 31714 raft_consensus.cc:3037] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:46.112553 31667 leader_election.cc:398] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Vote denied by peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607) with higher term. Message: Invalid argument: T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5.
I20260805 22:24:46.129789 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:46.130515 31490 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 4
I20260805 22:24:46.133415 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:46.134064 31500 raft_consensus.cc:1261] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:24:46.136528 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:46.138722 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:46.139194 31707 raft_consensus.cc:1261] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260805 22:24:46.140010 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:46.143477 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:46.144953 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:46.136003 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:46.147115 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:46.148187 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:46.148388 31444 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:46.148195 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:46.149997 31714 raft_consensus.cc:2454] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:46.136968 31500 raft_consensus.cc:1261] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260805 22:24:46.149848 31667 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:46.154619 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:46.156201 31667 leader_election.cc:398] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4.
I20260805 22:24:46.145845 31495 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:46.156800 31490 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:46.158056 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:46.159370 31444 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:46.160244 31444 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:46.160662 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:46.126226 32586 raft_consensus.cc:603] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:24:46.123417 2061 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:46.153549 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:46.166889 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:46.167393 31444 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:46.172456 32587 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:46.176275 31556 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:46.177492 31667 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:46.179374 31667 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:46.180523 31667 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:46.182354 31667 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:46.183238 2228 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:46.186278 31667 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:46.161038 2269 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:46.164131 2293 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:46.192298 31667 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:46.193769 31490 raft_consensus.cc:1261] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:24:46.173516 1920 raft_consensus.cc:2726] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:46.173261 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:46.158409 2276 consensus_queue.cc:1035] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.037s
I20260805 22:24:46.175652 2313 raft_consensus.cc:2781] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:46.188607 1952 raft_consensus.cc:2781] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:46.185665 1950 raft_consensus.cc:2726] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:46.185490 2137 raft_consensus.cc:2726] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:46.185676 2285 raft_consensus.cc:2726] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:46.165547 2254 consensus_queue.cc:1035] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20260805 22:24:46.192193 2076 raft_consensus.cc:2781] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:46.195402 2293 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:46.204461 31486 raft_consensus.cc:1261] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:46.198657 2303 raft_consensus.cc:2726] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:46.197706 2269 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:46.177932 2161 consensus_queue.cc:1035] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260805 22:24:46.200578 1958 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:46.233670 1958 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:46.235919 1958 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:46.185140 2222 consensus_queue.cc:1035] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.023s
I20260805 22:24:46.199602 2175 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:46.222764 2004 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260805 22:24:46.226756 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 51516.1us
I20260805 22:24:46.208688 2300 consensus_queue.cc:1035] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.022s
I20260805 22:24:46.178316 2267 consensus_queue.cc:1035] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.053s
I20260805 22:24:46.227437 2293 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:46.207659 2313 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:46.238811 2269 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:46.251771 2313 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:46.242969 31447 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:46.222713 1952 raft_consensus.cc:686] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
W20260805 22:24:46.233112 2327 log.cc:924] Time spent T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.078s user 0.000s sys 0.000s
I20260805 22:24:46.227299 1998 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:46.253062 1952 consensus_queue.cc:237] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:46.255002 1998 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:46.228370 2076 raft_consensus.cc:686] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:46.243597 1975 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:46.251289 2015 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:46.258116 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:46.260277 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:46.260506 31713 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:24:46.260648 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:46.262108 32529 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:46.244652 2004 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:46.264459 31447 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:46.264876 2004 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:46.264750 1975 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:46.267048 1975 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:46.268292 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:46.268091 2076 consensus_queue.cc:237] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:46.278363 1905 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:46.279667 31447 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:46.281100 1905 raft_consensus.cc:2726] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:46.282100 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:46.282804 31597 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:46.263589 2276 raft_consensus.cc:2726] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:46.306022 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:46.307664 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:46.308017 31706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58348 (request call id 416) took 1559 ms. Trace:
I20260805 22:24:46.308463 31706 rpcz_store.cc:276] 0805 22:24:44.748519 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:44.748679 (+ 160us) service_pool.cc:227] Handling call
0805 22:24:44.764717 (+ 16038us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:45.677225 (+912508us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:46.307757 (+630532us) spinlock_profiling.cc:243] Waited 900 ms on lock 0x7b54002b2ad4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:46.307951 (+ 194us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2069986560}
I20260805 22:24:46.309581 31603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1081) took 1679 ms. Trace:
I20260805 22:24:46.309939 31603 rpcz_store.cc:276] 0805 22:24:44.630315 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:44.632063 (+ 1748us) service_pool.cc:227] Handling call
0805 22:24:46.009724 (+1377661us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:46.010384 (+ 660us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:46.309399 (+299015us) spinlock_profiling.cc:243] Waited 1.38 s on lock 0x7b54002373d0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731067a09 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:46.309524 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3167733120}
I20260805 22:24:46.274156 2015 raft_consensus.cc:2781] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:46.311913 31602 raft_consensus.cc:1226] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
W20260805 22:24:46.321542 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 129327us
I20260805 22:24:46.323092 31667 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:46.323506 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:46.319577 2174 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:46.327072 1951 raft_consensus.cc:2726] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:46.327471 2174 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:46.316206 1998 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:46.330089 1998 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:46.331017 1951 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:46.331715 1951 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:46.332819 1998 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:46.333515 2244 consensus_queue.cc:1035] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:46.317200 2015 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:46.335444 2015 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:46.336308 1951 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:46.336786 1989 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:46.343686 31603 raft_consensus.cc:1261] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:46.344705 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:46.349963 31554 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:46.351286 2014 raft_consensus.cc:2726] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:46.353906 2174 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:46.354214 31495 raft_consensus.cc:1261] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:46.356877 1920 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:46.358758 1920 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:46.358892 2267 consensus_queue.cc:1046] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:24:46.362483 2004 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:46.363916 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:46.364666 31704 raft_consensus.cc:2379] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 3: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:46.366024 31709 raft_consensus.cc:1226] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:46.337363 1989 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:46.359999 2302 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:46.372119 31714 raft_consensus.cc:1226] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:46.375890 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:46.389321 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:46.397153 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:46.388614 31713 raft_consensus.cc:1261] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:46.402048 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:46.402745 31714 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:46.406570 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:46.407531 31713 raft_consensus.cc:2379] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 7: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:46.416065 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:46.416051 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:46.421006 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:46.423749 31708 raft_consensus.cc:1261] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260805 22:24:46.431145 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:46.434248 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:46.435674 31554 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.602s (SENT) [suppressed 14 similar messages]
I20260805 22:24:46.437151 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:46.437992 31704 raft_consensus.cc:2379] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 3: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:46.438794 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:46.439457 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:46.387143 1905 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:46.386140 2004 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:46.444164 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:46.444774 31501 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:46.447049 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:46.413529 2270 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:46.389498 1989 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:46.449420 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:46.456919 31495 raft_consensus.cc:2379] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 4: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:46.446585 1920 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:46.457854 31494 raft_consensus.cc:1261] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:24:46.435324 1951 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:46.464231 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:46.466184 31494 raft_consensus.cc:2454] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:46.463096 31495 raft_consensus.cc:1261] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260805 22:24:46.458881 31714 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:46.448117 2244 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:46.386212 2302 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:46.468706 32594 raft_consensus.cc:1261] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260805 22:24:46.471901 31494 raft_consensus.cc:1226] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:46.455173 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:46.478745 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:46.462150 31501 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:46.470238 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:46.458254 31597 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:46.468034 2164 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:46.488721 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:46.478798 2244 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:46.458787 2270 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:46.453099 1905 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:46.569906 2164 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:46.572090 2164 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:24:46.575857 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 85230.9us
I20260805 22:24:46.610044 2276 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:46.610692 2302 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:46.471161 1951 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:46.612898 1951 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:46.484320 2339 consensus_queue.cc:1035] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.130s
W20260805 22:24:46.614463 2355 log.cc:924] Time spent T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.125s user 0.000s sys 0.000s
I20260805 22:24:46.619243 2015 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:46.619436 2269 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:46.621940 2015 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:46.624114 2270 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:46.491899 2348 consensus_queue.cc:1046] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.140s
I20260805 22:24:46.489952 2161 consensus_queue.cc:1035] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.140s
I20260805 22:24:46.631577 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:46.678329 32594 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:46.630095 31499 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
W20260805 22:24:46.566588 2351 log.cc:924] Time spent T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.085s user 0.001s sys 0.000s
I20260805 22:24:46.630445 2340 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:46.674788 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:46.699693 2359 log.cc:924] Time spent T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.124s user 0.000s sys 0.001s
I20260805 22:24:46.616987 2313 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:46.626214 2244 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:46.610687 2276 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:46.701100 2269 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:46.701026 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:46.706513 31668 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:24:46.708153 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 272483us
W20260805 22:24:46.742034 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 292237us
I20260805 22:24:46.742486 32594 raft_consensus.cc:1261] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260805 22:24:47.032719 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:46.825546 2181 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:47.033931 2181 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.035915 2181 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:46.822264 31493 raft_consensus.cc:1226] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:46.830550 31707 raft_consensus.cc:1226] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 5. Current term is 6. Ops: []
W20260805 22:24:46.902776 2357 log.cc:924] Time spent T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.102s user 0.001s sys 0.000s
I20260805 22:24:46.831084 31502 raft_consensus.cc:1226] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260805 22:24:47.051705 31602 raft_consensus.cc:1261] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260805 22:24:47.053774 31444 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:47.054055 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:47.054422 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 347978us
I20260805 22:24:47.057813 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:47.059648 31490 raft_consensus.cc:1261] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:47.059923 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:47.061059 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:47.061825 31501 raft_consensus.cc:2379] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
W20260805 22:24:46.856634 2293 consensus_meta.cc:287] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.592s user 0.010s sys 0.005s
I20260805 22:24:47.043603 2269 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:47.066605 31357 catalog_manager.cc:5495] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:47.068295 31444 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:46.823134 2362 log.cc:924] Time spent T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.148s user 0.000s sys 0.000s
I20260805 22:24:47.080976 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:47.071918 32529 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:47.076306 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:47.084229 31444 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:47.071029 2125 raft_consensus.cc:2726] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:46.748740 32579 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:47.095249 32529 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:47.092330 2337 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:24:47.092607 2080 raft_consensus.cc:2726] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:47.097115 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:47.097671 2193 consensus_queue.cc:1035] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:24:47.097776 1998 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:47.100220 2337 consensus_queue.cc:1035] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:47.085543 31357 catalog_manager.cc:5495] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 4 to 5, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195), now has a pending config: VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193), VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194), VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195), VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 5 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } } pending_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:47.097924 31600 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:47.105057 2015 raft_consensus.cc:2726] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:47.104760 31357 catalog_manager.cc:5495] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 1 to 3, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:24:47.105645 2080 consensus_queue.cc:1035] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260805 22:24:47.105885 32529 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:47.107019 32529 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:47.106523 31357 catalog_manager.cc:5495] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 3 to 4, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 4 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:47.107439 2080 raft_consensus.cc:2726] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:47.107890 2309 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:47.108646 2348 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:47.109239 1951 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:47.109089 2348 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.109735 1951 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:47.111443 2348 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:47.111828 1951 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:47.112317 2222 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:47.112962 2222 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:47.115068 2222 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:47.105597 2365 consensus_queue.cc:1046] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.013s
I20260805 22:24:47.119556 2365 raft_consensus.cc:3032] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:47.119886 2365 raft_consensus.cc:729] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:47.120683 2365 consensus_queue.cc:260] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.121978 2365 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:47.084443 2293 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:47.124836 2293 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:47.074148 2078 raft_consensus.cc:2781] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:47.125506 2078 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:47.125926 2078 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
W20260805 22:24:47.130156 31359 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34400 (request call id 284) took 11811 ms (client timeout 15000 ms). Trace:
I20260805 22:24:47.131309 31668 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:47.132669 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:24:47.141472 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 99390.6us
I20260805 22:24:46.739140 2340 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:47.150825 32529 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:47.160586 31707 raft_consensus.cc:1226] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260805 22:24:47.189998 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:47.190773 32587 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:47.130100 2238 consensus_queue.cc:1046] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260805 22:24:47.143682 2293 raft_consensus.cc:2781] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260805 22:24:47.205556 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 54841.4us
W20260805 22:24:47.144802 31359 rpcz_store.cc:269] 0805 22:24:35.319243 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:35.436415 (+117172us) service_pool.cc:227] Handling call
0805 22:24:47.130061 (+11693646us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0805 22:24:42.971606 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0805 22:24:43.051292 (+ 79686us) write_op.cc:432] Acquiring schema lock in shared mode
0805 22:24:43.051327 (+ 35us) write_op.cc:435] Acquired schema lock
0805 22:24:43.051347 (+ 20us) tablet.cc:660] Decoding operations
0805 22:24:43.052250 (+ 903us) write_op.cc:620] Acquiring the partition lock for write op
0805 22:24:43.052306 (+ 56us) write_op.cc:641] Partition lock acquired for write op
0805 22:24:43.052330 (+ 24us) tablet.cc:683] Acquiring locks for 9 operations
0805 22:24:43.052984 (+ 654us) tablet.cc:699] Row locks acquired
0805 22:24:43.053002 (+ 18us) write_op.cc:260] PREPARE: finished
0805 22:24:43.053141 (+ 139us) write_op.cc:270] Start()
0805 22:24:43.053260 (+ 119us) write_op.cc:276] Timestamp: P: 1785968683053096 usec, L: 0
0805 22:24:43.053279 (+ 19us) op_driver.cc:331] REPLICATION: starting
0805 22:24:43.167468 (+114189us) log.cc:842] Serialized 3641 byte log entry
0805 22:24:45.223434 (+2055966us) spinlock_profiling.cc:243] Waited 9.38 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:24:45.727417 (+503983us) op_driver.cc:444] REPLICATION: finished
0805 22:24:45.971713 (+244296us) write_op.cc:301] APPLY: starting
0805 22:24:45.974689 (+ 2976us) tablet.cc:1365] starting BulkCheckPresence
0805 22:24:46.044410 (+ 69721us) tablet.cc:1368] finished BulkCheckPresence
0805 22:24:46.053533 (+ 9123us) tablet.cc:1370] starting ApplyRowOperation cycle
0805 22:24:46.054579 (+ 1046us) tablet.cc:1381] finished ApplyRowOperation cycle
0805 22:24:46.054838 (+ 259us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0805 22:24:46.054887 (+ 49us) write_op.cc:312] APPLY: finished
0805 22:24:46.055217 (+ 330us) log.cc:842] Serialized 71 byte log entry
0805 22:24:46.055391 (+ 174us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:46.055558 (+ 167us) write_op.cc:454] Released schema lock
0805 22:24:46.055609 (+ 51us) write_op.cc:341] FINISH: Updating metrics
0805 22:24:46.055928 (+ 319us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:46.055955 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":838880,"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":183345,"apply.run_cpu_time_us":3397,"apply.run_wall_time_us":116940,"num_ops":9,"prepare.queue_time_us":127159,"prepare.run_cpu_time_us":57112,"prepare.run_wall_time_us":566517,"raft.queue_time_us":304333,"raft.run_cpu_time_us":11035,"raft.run_wall_time_us":160864,"replication_time_us":2670614,"spinlock_wait_cycles":21578496,"thread_start_us":720764,"threads_started":4,"wal-append.queue_time_us":348696,"wal-append.run_cpu_time_us":41900,"wal-append.run_wall_time_us":3532452}]]}
I20260805 22:24:47.209014 2309 raft_consensus.cc:2726] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:47.201504 1952 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:47.215675 2340 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:47.217417 2238 raft_consensus.cc:3032] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:47.087780 2269 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:47.224265 31668 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:47.228363 2078 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.201013 32584 raft_consensus.cc:1261] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260805 22:24:47.153236 2224 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:47.251823 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:47.254360 31715 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:47.233880 1952 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.237593 2293 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:47.238648 2238 raft_consensus.cc:729] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:47.250818 2340 raft_consensus.cc:2781] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:47.253600 2368 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:47.261582 2015 consensus_queue.cc:1046] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.009s
I20260805 22:24:47.263852 2293 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:47.271322 2340 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:47.259807 2224 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.292167 2340 raft_consensus.cc:3037] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:47.296586 32581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57382 (request call id 581) took 1184 ms. Trace:
I20260805 22:24:47.279345 2369 consensus_queue.cc:1035] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.025s
I20260805 22:24:47.275249 2368 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.314393 32529 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:47.322116 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:47.322683 31667 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
W20260805 22:24:47.330343 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 187288us
I20260805 22:24:47.311094 2269 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:47.337046 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:47.339601 31444 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:47.332880 2254 raft_consensus.cc:2781] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:47.340181 2254 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:47.340560 2254 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:47.340807 2300 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:24:47.341230 2300 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:47.341579 2300 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:47.312777 2078 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:47.308732 1741 consensus_queue.cc:1046] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.024s
I20260805 22:24:47.360677 2368 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:47.362174 2234 raft_consensus.cc:3032] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:47.362607 2234 raft_consensus.cc:729] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:47.363418 2234 consensus_queue.cc:260] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.364460 2234 raft_consensus.cc:3037] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:47.364131 2254 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.365444 2224 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:47.342794 2222 consensus_queue.cc:1035] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.032s
I20260805 22:24:47.310685 32581 rpcz_store.cc:276] 0805 22:24:46.112049 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:46.122147 (+ 10098us) service_pool.cc:227] Handling call
0805 22:24:47.253238 (+1131091us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b5800005198. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108bce 000000000034aea9 00007f2735f34a30 00007f2735ffa7b4 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:24:47.262050 (+ 8812us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:47.271805 (+ 9755us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:47.273420 (+ 1615us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:47.276245 (+ 2825us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:24:47.278592 (+ 2347us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:47.283803 (+ 5211us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:47.294783 (+ 10980us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":23226112}
I20260805 22:24:47.339490 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:47.330894 2164 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:47.299074 2238 consensus_queue.cc:260] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.352458 31554 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:47.318945 1989 raft_consensus.cc:2781] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:47.354856 1952 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:47.338609 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:47.373926 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:47.377929 31554 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:47.393098 31554 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:47.394433 31554 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:47.395884 31554 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:47.373016 2324 raft_consensus.cc:2781] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:47.372392 2158 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:47.372757 1989 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:47.371168 2254 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:47.392300 1952 raft_consensus.cc:2726] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260805 22:24:47.428216 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 100848us
I20260805 22:24:47.410360 2377 raft_consensus.cc:2726] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:47.411249 2378 raft_consensus.cc:2726] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:47.430028 32587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:60186
I20260805 22:24:47.417788 2324 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:47.422648 2222 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:47.412035 1754 consensus_queue.cc:1035] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.021s
I20260805 22:24:47.542069 31705 raft_consensus.cc:1226] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260805 22:24:47.542714 1754 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:47.544061 1754 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.437786 1952 raft_consensus.cc:2726] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:47.546847 1952 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:47.547691 1952 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:47.425935 1989 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:47.553359 2224 consensus_queue.cc:1046] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:24:47.556597 31493 raft_consensus.cc:1226] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260805 22:24:47.558883 1741 consensus_queue.cc:1046] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260805 22:24:47.487723 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:47.499779 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:47.500275 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:47.565268 31708 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:24:47.525738 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:47.498281 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:47.532055 32594 raft_consensus.cc:1226] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 5. Current term is 6. Ops: []
W20260805 22:24:47.570158 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 67294us
I20260805 22:24:47.535014 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:47.536629 2064 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:47.372184 2293 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.441685 2222 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:47.571552 31667 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:47.371696 2164 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:47.573757 31667 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:47.570343 31713 raft_consensus.cc:2454] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:47.412346 2303 consensus_queue.cc:1035] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.023s
I20260805 22:24:47.534111 1998 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:47.439545 2324 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:47.377276 2340 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.601815 2222 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:47.531111 32585 raft_consensus.cc:1261] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260805 22:24:47.574249 2004 raft_consensus.cc:2726] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:24:47.604286 2340 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:47.571949 2064 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.613648 2064 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:47.614039 2300 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:47.614645 1952 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:47.615576 1952 consensus_queue.cc:1035] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260805 22:24:47.620692 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 47150.1us
I20260805 22:24:47.621013 1975 raft_consensus.cc:2726] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:47.622404 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:47.623766 2164 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:24:47.624388 2388 log.cc:924] Time spent T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.180s user 0.000s sys 0.001s
I20260805 22:24:47.628897 2383 consensus_queue.cc:1046] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260805 22:24:47.630236 2383 raft_consensus.cc:3032] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:24:47.630708 2383 raft_consensus.cc:729] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:47.631861 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:47.632011 2383 consensus_queue.cc:260] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.632747 32579 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:24:47.633882 2383 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:47.606565 1998 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.636544 2293 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:47.636870 1998 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:47.641067 31447 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:47.629412 31715 raft_consensus.cc:1226] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:47.642220 2369 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:47.642693 2369 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:47.643195 2369 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:47.646117 31667 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:47.646930 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:47.648572 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:47.664691 32529 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:47.653088 1905 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:47.653131 2078 raft_consensus.cc:2726] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260805 22:24:47.692441 2374 log.cc:924] Time spent T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.355s user 0.004s sys 0.003s
W20260805 22:24:47.703998 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 307996us
I20260805 22:24:47.705721 1952 consensus_queue.cc:1035] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:24:47.661469 1998 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:47.705897 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:47.710180 31554 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:47.718673 31711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 2147) took 1193 ms. Trace:
I20260805 22:24:47.742127 31709 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:47.743209 2188 raft_consensus.cc:3032] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:47.743613 31597 raft_consensus.cc:1261] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260805 22:24:47.744993 31706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1718) took 1388 ms. Trace:
I20260805 22:24:47.745486 31706 rpcz_store.cc:276] 0805 22:24:46.356584 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:46.360280 (+ 3696us) service_pool.cc:227] Handling call
0805 22:24:46.824380 (+464100us) spinlock_profiling.cc:243] Waited 3.89 ms on lock 0x7b58002d0698. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:24:46.824589 (+ 209us) raft_consensus.cc:1457] Updating replica for 2 ops
0805 22:24:47.044500 (+219911us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:47.044531 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0805 22:24:47.045044 (+ 513us) log.cc:842] Serialized 52 byte log entry
0805 22:24:47.143295 (+ 98251us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:24:47.143330 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:47.143394 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:47.674442 (+531048us) spinlock_profiling.cc:243] Waited 15.5 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:24:47.744492 (+ 70050us) raft_consensus.cc:1671] finished
0805 22:24:47.744518 (+ 26us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:47.744785 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":219406,"raft.queue_time_us":220694,"raft.run_cpu_time_us":101,"raft.run_wall_time_us":101,"spinlock_wait_cycles":44582144,"thread_start_us":97912,"threads_started":1,"wal-append.queue_time_us":262866}
I20260805 22:24:47.743677 2188 raft_consensus.cc:729] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:47.717348 2080 raft_consensus.cc:2726] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:47.750085 31709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1731) took 1317 ms. Trace:
I20260805 22:24:47.726531 2300 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:47.696110 1989 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.726704 2313 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:47.750527 31709 rpcz_store.cc:276] 0805 22:24:46.430355 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:46.430723 (+ 368us) service_pool.cc:227] Handling call
0805 22:24:47.747369 (+1316646us) spinlock_profiling.cc:243] Waited 1.31 s on lock 0x7b54001e3554. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:47.747527 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3006917888}
I20260805 22:24:47.721676 2276 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:47.718616 32584 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:47.756317 1998 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:47.758692 2188 consensus_queue.cc:260] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.760331 2365 raft_consensus.cc:3032] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:24:47.764918 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:47.765357 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:47.766867 31603 raft_consensus.cc:3032] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:47.767333 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:47.747942 2228 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:24:47.775101 2381 log.cc:924] Time spent T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.427s user 0.000s sys 0.002s
I20260805 22:24:47.775875 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:47.776213 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:47.795588 31596 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:47.796157 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:47.789517 2365 raft_consensus.cc:729] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:47.798540 2365 consensus_queue.cc:260] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.799837 1998 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:47.800006 2365 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:47.800498 2394 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:24:47.801108 2394 raft_consensus.cc:686] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 6 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:47.801672 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:47.802608 31490 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:47.802323 2394 consensus_queue.cc:237] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:47.803624 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:47.804556 31490 raft_consensus.cc:2379] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 6: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:47.804873 32583 raft_consensus.cc:2379] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 5: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:47.777153 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:47.778913 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:47.806735 31499 raft_consensus.cc:2379] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 8: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:47.793923 31603 raft_consensus.cc:729] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:47.777073 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:47.781446 31502 raft_consensus.cc:1226] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260805 22:24:47.735221 31711 rpcz_store.cc:276] 0805 22:24:46.465009 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:46.466724 (+ 1715us) service_pool.cc:227] Handling call
0805 22:24:46.468495 (+ 1771us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:47.043472 (+574977us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:47.043499 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:47.043944 (+ 445us) log.cc:842] Serialized 26 byte log entry
0805 22:24:47.320779 (+276835us) raft_consensus.cc:1631] Marking committed up to 2
0805 22:24:47.320816 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:47.644859 (+324043us) spinlock_profiling.cc:243] Waited 102 ms on lock 0x7b54000339d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:47.644930 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:47.645023 (+ 93us) raft_consensus.cc:1671] finished
0805 22:24:47.645038 (+ 15us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:47.658578 (+ 13540us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":472114,"raft.queue_time_us":472254,"raft.run_cpu_time_us":92,"raft.run_wall_time_us":92,"spinlock_wait_cycles":235487104,"thread_start_us":241259,"threads_started":1,"wal-append.queue_time_us":322459}
I20260805 22:24:47.752967 1952 consensus_queue.cc:1035] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.032s
I20260805 22:24:47.804643 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:47.812180 31494 raft_consensus.cc:2379] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:47.779304 1754 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:47.812536 31500 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:47.811537 31603 consensus_queue.cc:260] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:47.798605 2188 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:47.817629 31554 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:47.826463 1989 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:47.827579 2238 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:47.824533 31596 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:24:47.852644 2387 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:47.853353 2387 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:47.855548 2387 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:47.857532 31667 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:47.801575 2228 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:47.857247 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:47.861951 2293 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:47.875540 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:47.879596 31603 raft_consensus.cc:3037] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
W20260805 22:24:47.878435 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 88802.2us
I20260805 22:24:47.863675 31444 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:47.866874 2293 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:47.881067 2293 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:47.881592 31668 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:47.881748 1754 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:47.882431 1754 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.885007 31602 raft_consensus.cc:1226] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4. Ops: [3.6-3.6]
I20260805 22:24:47.886175 2377 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:47.886446 2193 raft_consensus.cc:2781] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:47.887109 2377 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:47.882386 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 73043.2us
I20260805 22:24:47.890480 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:47.891296 31709 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:47.892932 2387 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:47.894042 2387 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:47.867504 2313 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:47.894443 2387 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:47.894922 2377 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:47.894661 2313 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.896054 1754 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:47.887208 2193 raft_consensus.cc:686] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:47.896819 2313 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:47.897393 2324 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:47.897900 2193 consensus_queue.cc:237] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.899660 2324 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:47.901690 31604 raft_consensus.cc:1226] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:47.902348 2369 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:47.904647 2369 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:47.902295 2293 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:47.906612 31602 raft_consensus.cc:1261] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260805 22:24:47.907281 31495 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:47.907756 2293 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:47.890514 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:47.911749 31709 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:24:47.925521 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:47.927280 32587 raft_consensus.cc:595] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:47.927752 32587 raft_consensus.cc:603] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:24:47.930083 2369 raft_consensus.cc:2726] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260805 22:24:47.932385 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:47.928221 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:47.933326 31594 raft_consensus.cc:2379] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 7: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:47.947250 31602 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
W20260805 22:24:47.948544 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 85084us
I20260805 22:24:47.975404 31603 raft_consensus.cc:1261] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
W20260805 22:24:48.047576 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 111365us
I20260805 22:24:48.048410 32529 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:48.071961 2224 raft_consensus.cc:2781] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:48.072584 2224 raft_consensus.cc:686] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:48.102146 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:48.104298 2224 consensus_queue.cc:237] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.105082 2337 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:48.112556 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:48.112900 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:48.113675 31502 raft_consensus.cc:2362] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6. Re-sending same reply.
I20260805 22:24:48.112629 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:48.115547 31486 raft_consensus.cc:2379] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:48.116202 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:48.116768 31444 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:48.116986 2378 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:48.117574 1952 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:48.118270 1952 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.116931 31709 raft_consensus.cc:2379] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 5: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:48.120294 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:48.123379 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:48.130082 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:48.142554 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:48.149910 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:48.149247 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:48.156622 2064 raft_consensus.cc:2781] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:48.156790 1952 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:48.157707 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:48.157856 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:48.157568 2378 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.158684 31706 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
W20260805 22:24:48.160676 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 44266.6us
I20260805 22:24:48.160907 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:48.162058 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:48.163520 31444 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:48.163729 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:48.174011 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:48.165500 31707 raft_consensus.cc:1226] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:48.157125 2064 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:48.175076 2064 raft_consensus.cc:3037] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:48.175354 31500 raft_consensus.cc:1226] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 5. Current term is 6. Ops: []
I20260805 22:24:48.160931 2337 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.165915 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:48.169137 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:48.169489 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:48.170260 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:48.138360 2387 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:48.186949 31601 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:48.188071 2387 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:48.174427 2378 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:48.183048 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:48.186198 31499 raft_consensus.cc:2379] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 7: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:48.198094 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:48.198400 31603 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:48.198908 31604 raft_consensus.cc:2379] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 7: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:48.199371 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:48.199836 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:48.200555 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:48.184250 2313 raft_consensus.cc:2781] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:24:48.201711 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:48.201228 31603 raft_consensus.cc:2379] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:48.178187 2337 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:48.206246 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:48.212026 31601 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
I20260805 22:24:48.224144 32587 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:24:48.205034 2387 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:48.229667 31447 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:48.205293 2313 raft_consensus.cc:686] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:48.239178 2378 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:48.212065 31709 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56246
I20260805 22:24:48.250289 31709 raft_consensus.cc:595] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Received request to transfer leadership
I20260805 22:24:48.244891 1998 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:24:48.257951 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:48.265005 2339 consensus_queue.cc:1035] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260805 22:24:48.256222 2378 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.270628 2378 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:48.271401 2293 raft_consensus.cc:979] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385: : Instructing follower 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:48.271661 31604 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 7.
I20260805 22:24:48.272075 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:48.273847 2313 consensus_queue.cc:237] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.276108 31707 raft_consensus.cc:1226] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:48.271908 2293 raft_consensus.cc:1067] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Signalling peer 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:24:48.279572 32583 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:57366
I20260805 22:24:48.280246 32583 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:48.280606 32583 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:48.249538 2387 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.282143 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:48.283653 2387 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:48.289876 1998 raft_consensus.cc:686] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:48.297102 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:48.297640 2228 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:48.297807 31493 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:48.301807 1998 consensus_queue.cc:237] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.307309 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:48.290144 32583 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.312588 32585 raft_consensus.cc:1261] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260805 22:24:48.314510 2228 consensus_queue.cc:1035] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260805 22:24:48.315861 31709 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56246
I20260805 22:24:48.316499 31709 raft_consensus.cc:595] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Received request to transfer leadership
W20260805 22:24:48.305259 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 425111us
W20260805 22:24:48.325542 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 57684.3us
I20260805 22:24:48.327876 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:48.331415 32583 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:48.339280 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:48.340795 31493 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:48.341408 31609 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1115) took 1005 ms. Trace:
I20260805 22:24:48.341847 31609 rpcz_store.cc:276] 0805 22:24:47.335719 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:47.335849 (+ 130us) service_pool.cc:227] Handling call
0805 22:24:47.768118 (+432269us) spinlock_profiling.cc:243] Waited 1.26 ms on lock 0x7b5800004598. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:24:47.768524 (+ 406us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:47.876938 (+108414us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:47.876957 (+ 19us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:47.935908 (+ 58951us) log.cc:842] Serialized 26 byte log entry
0805 22:24:48.152509 (+216601us) raft_consensus.cc:1631] Marking committed up to 2
0805 22:24:48.152547 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:48.152612 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:48.340958 (+188346us) raft_consensus.cc:1671] finished
0805 22:24:48.340977 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:48.341208 (+ 231us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1770,"raft.run_cpu_time_us":138,"raft.run_wall_time_us":137,"spinlock_wait_cycles":2894208,"thread_start_us":205729,"threads_started":2,"wal-append.queue_time_us":284780}
I20260805 22:24:48.347136 31667 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:48.349797 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:48.352238 2078 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:48.354804 2078 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:48.355170 2078 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:48.371177 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:48.345459 2064 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:48.374384 32587 raft_consensus.cc:2396] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260805 22:24:48.375119 31667 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:48.352882 2228 consensus_queue.cc:1046] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20260805 22:24:48.387482 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:48.408704 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:48.409941 31490 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:48.416709 31667 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:48.388480 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:48.424489 2365 consensus_queue.cc:1046] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:24:48.425966 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:48.429045 31709 raft_consensus.cc:3032] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:24:48.429350 31709 raft_consensus.cc:729] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:48.429741 31601 raft_consensus.cc:1226] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260805 22:24:48.430457 31709 consensus_queue.cc:260] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.444330 32585 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:60186
I20260805 22:24:48.464211 31709 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:48.446120 2193 raft_consensus.cc:2781] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260805 22:24:48.472244 31339 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.LeaderStepDown -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 140989us
I20260805 22:24:48.472288 32585 raft_consensus.cc:595] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:48.476575 2254 raft_consensus.cc:2726] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:48.477005 2064 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:48.447981 31600 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:48.487800 2078 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:48.473304 2193 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260805 22:24:48.493559 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 249285us
I20260805 22:24:48.485205 31604 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
I20260805 22:24:48.503391 31490 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
W20260805 22:24:48.505507 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 110716us
I20260805 22:24:48.507041 31667 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:48.511663 31444 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:48.508920 2193 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:48.514129 2254 raft_consensus.cc:2726] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:48.520660 31713 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56246
I20260805 22:24:48.523017 2293 raft_consensus.cc:3032] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:24:48.523465 2293 raft_consensus.cc:729] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:48.524302 2293 consensus_queue.cc:260] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:48.535048 2324 consensus_queue.cc:1046] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:48.536134 2324 raft_consensus.cc:3032] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:48.536445 2324 raft_consensus.cc:729] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:48.536991 31357 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34394 (request call id 295) took 10990 ms. Trace:
I20260805 22:24:48.539592 31554 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:48.538111 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
W20260805 22:24:48.540371 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 119445us
I20260805 22:24:48.540412 31594 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:48.540797 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:48.541594 31604 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:48.502413 2300 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:48.537992 31351 catalog_manager.cc:5495] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 4 to 5, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 5 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:48.551152 31554 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:48.555127 2387 raft_consensus.cc:2726] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:48.556102 31554 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:48.557421 2383 consensus_queue.cc:1035] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:48.557155 31351 catalog_manager.cc:5495] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index 2 to 5, 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) changed from NON_VOTER to VOTER, no longer has a pending config: VOTER f8e9f4c756da4203917c464e2b179994 (127.29.21.193), VOTER 516cb07ce3064ccbbca35e912d62816c (127.29.21.194), VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195), VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 5 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:48.558893 2383 raft_consensus.cc:2726] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:48.559530 31351 catalog_manager.cc:5495] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 3 to 4, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 4 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:24:48.551631 2378 raft_consensus.cc:2726] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:48.547283 2276 raft_consensus.cc:2781] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:48.561403 2276 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:48.561669 31351 catalog_manager.cc:5495] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 4 to 6, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 6 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:48.537370 2324 consensus_queue.cc:260] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:48.550992 2437 consensus_queue.cc:1035] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:48.567598 2369 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:48.557642 2188 raft_consensus.cc:3032] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Stepping down as leader of term 3
W20260805 22:24:48.566371 2426 log.cc:924] Time spent T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.069s user 0.002s sys 0.000s
I20260805 22:24:48.563651 2324 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:48.567473 2133 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:48.570220 2133 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.572247 2133 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:48.572960 2064 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:48.547816 2293 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:48.568221 2369 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.561764 2276 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:48.573725 2064 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:48.569885 2188 raft_consensus.cc:729] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:48.575655 2064 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:48.575323 2369 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:48.567436 1952 consensus_queue.cc:1035] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:48.537372 31357 rpcz_store.cc:276] 0805 22:24:37.546186 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:37.546976 (+ 790us) service_pool.cc:227] Handling call
0805 22:24:48.536749 (+10989773us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0805 22:24:47.361638 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0805 22:24:47.362156 (+ 518us) write_op.cc:432] Acquiring schema lock in shared mode
0805 22:24:47.362187 (+ 31us) write_op.cc:435] Acquired schema lock
0805 22:24:47.362204 (+ 17us) tablet.cc:660] Decoding operations
0805 22:24:47.362785 (+ 581us) write_op.cc:620] Acquiring the partition lock for write op
0805 22:24:47.362844 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0805 22:24:47.362869 (+ 25us) tablet.cc:683] Acquiring locks for 4 operations
0805 22:24:47.363245 (+ 376us) tablet.cc:699] Row locks acquired
0805 22:24:47.363263 (+ 18us) write_op.cc:260] PREPARE: finished
0805 22:24:47.363395 (+ 132us) write_op.cc:270] Start()
0805 22:24:47.363500 (+ 105us) write_op.cc:276] Timestamp: P: 1785968687363353 usec, L: 0
0805 22:24:47.363521 (+ 21us) op_driver.cc:331] REPLICATION: starting
0805 22:24:47.364104 (+ 583us) log.cc:842] Serialized 2010 byte log entry
0805 22:24:47.722170 (+358066us) op_driver.cc:444] REPLICATION: finished
0805 22:24:47.857552 (+135382us) write_op.cc:301] APPLY: starting
0805 22:24:47.857694 (+ 142us) tablet.cc:1365] starting BulkCheckPresence
0805 22:24:47.857917 (+ 223us) tablet.cc:1368] finished BulkCheckPresence
0805 22:24:47.858079 (+ 162us) tablet.cc:1370] starting ApplyRowOperation cycle
0805 22:24:47.858972 (+ 893us) tablet.cc:1381] finished ApplyRowOperation cycle
0805 22:24:47.859313 (+ 341us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0805 22:24:47.859367 (+ 54us) write_op.cc:312] APPLY: finished
0805 22:24:47.859759 (+ 392us) log.cc:842] Serialized 41 byte log entry
0805 22:24:47.919183 (+ 59424us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:47.946329 (+ 27146us) write_op.cc:454] Released schema lock
0805 22:24:47.946390 (+ 61us) write_op.cc:341] FINISH: Updating metrics
0805 22:24:47.946786 (+ 396us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:47.946820 (+ 34us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":34011,"reported_tablets":24,"child_traces":[["op",{"apply.queue_time_us":135029,"apply.run_cpu_time_us":11538,"apply.run_wall_time_us":239312,"mutex_wait_us":4881,"num_ops":4,"prepare.queue_time_us":243157,"prepare.run_cpu_time_us":25070,"prepare.run_wall_time_us":75006,"raft.queue_time_us":271500,"raft.run_cpu_time_us":18607,"raft.run_wall_time_us":105281,"replication_time_us":358502,"thread_start_us":638958,"threads_started":4,"wal-append.queue_time_us":81458}]]}
I20260805 22:24:48.556546 2193 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:48.563289 31351 catalog_manager.cc:5495] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 4 to 5, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 5 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:24:48.581795 2193 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:48.582319 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:48.583650 2078 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:48.584270 31351 catalog_manager.cc:5495] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 3 to 4, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 4 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:24:48.593669 32583 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:60186
I20260805 22:24:48.590476 31351 catalog_manager.cc:5495] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 1 to 5, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 5 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:48.594400 32583 raft_consensus.cc:595] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:48.595280 2300 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:48.569034 31709 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
I20260805 22:24:48.597007 2133 raft_consensus.cc:979] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf: : Instructing follower feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:48.597468 2133 raft_consensus.cc:1067] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Signalling peer feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:48.597641 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:48.597858 31713 raft_consensus.cc:595] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Received request to transfer leadership
I20260805 22:24:48.598917 31713 raft_consensus.cc:603] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:24:48.600411 31490 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:24:48.601002 31490 raft_consensus.cc:595] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:48.601296 31490 raft_consensus.cc:603] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:24:48.575839 2188 consensus_queue.cc:260] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.602912 2188 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:48.603502 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
W20260805 22:24:48.602252 31423 auto_rebalancer.cc:663] Could not move replica: Illegal state: not currently leader
W20260805 22:24:48.604096 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Illegal state: not currently leader
I20260805 22:24:48.606411 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:24:48.607311 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:48.610368 2387 consensus_queue.cc:1046] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:48.611392 1952 raft_consensus.cc:3032] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:24:48.611759 1952 raft_consensus.cc:729] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:48.612697 1952 consensus_queue.cc:260] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:48.615442 31594 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
I20260805 22:24:48.618731 2300 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:48.620083 31554 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:48.624771 1952 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:48.569288 31444 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
W20260805 22:24:48.627985 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 58778.6us
W20260805 22:24:48.628226 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 118633us
I20260805 22:24:48.632298 31490 raft_consensus.cc:1226] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 5. Current term is 6. Ops: []
I20260805 22:24:48.581130 31604 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:24:48.632834 2378 raft_consensus.cc:2781] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:48.635735 2378 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:48.634047 2300 raft_consensus.cc:2726] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:48.636885 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:48.636886 31444 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:48.603042 31351 catalog_manager.cc:5495] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 reported cstate change: config changed from index 4 to 5, VOTER feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) evicted. New cstate: current_term: 2 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:48.639508 31444 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:48.639693 31554 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:48.636183 2378 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:48.643951 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:48.648799 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:48.649880 31500 raft_consensus.cc:1261] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260805 22:24:48.643431 2300 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:48.654815 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:48.657426 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 103359us
I20260805 22:24:48.657357 31351 catalog_manager.cc:5495] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 3 to 4, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 4 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:48.643121 2369 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:48.663409 31554 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:48.664279 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:48.627877 2276 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:48.664846 31500 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:48.665594 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:48.661146 1754 raft_consensus.cc:2726] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:48.666464 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:48.666606 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:48.666745 2369 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.654691 2064 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election won for term 8
I20260805 22:24:48.668213 2064 raft_consensus.cc:686] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 8 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:48.668780 2369 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:48.671252 31709 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:58348
I20260805 22:24:48.671826 31709 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:48.672179 31709 raft_consensus.cc:3037] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:48.677465 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:48.681432 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:48.677623 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:48.672266 2293 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:48.695019 32583 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:48.695411 2293 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:48.669173 2064 consensus_queue.cc:237] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:48.692813 31597 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:48.671233 2300 consensus_queue.cc:1046] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260805 22:24:48.698263 2300 raft_consensus.cc:3032] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:48.698650 2300 raft_consensus.cc:729] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:48.699606 2300 consensus_queue.cc:260] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.701063 2300 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:48.680821 2394 raft_consensus.cc:2726] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:48.702482 31709 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.678560 2276 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:48.678892 31490 raft_consensus.cc:2379] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:48.698575 31447 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:48.670444 31351 catalog_manager.cc:5495] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:48.705370 31500 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:24:48.705525 31709 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:48.713658 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:48.673820 31554 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:48.703743 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:48.718484 31604 raft_consensus.cc:2379] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:48.711983 31351 catalog_manager.cc:5495] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 3 to 4, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 4 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:48.662063 2452 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:48.737406 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:48.737663 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:48.728892 31444 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:48.723651 31351 catalog_manager.cc:5495] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 2 to 3, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:48.740868 31501 raft_consensus.cc:1226] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:48.745507 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:48.746991 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:48.747056 31706 raft_consensus.cc:1261] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260805 22:24:48.748389 31704 raft_consensus.cc:2379] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:48.721480 2394 raft_consensus.cc:2781] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:48.748669 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:48.749332 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 122307us
I20260805 22:24:48.749832 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:48.750205 31609 raft_consensus.cc:2379] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:48.711874 2293 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:48.718111 2193 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:48.745667 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:48.750907 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:48.717667 2437 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:48.690716 2378 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:48.753659 31492 raft_consensus.cc:2379] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:48.754007 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:48.754518 31715 raft_consensus.cc:1261] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:24:48.757002 31707 raft_consensus.cc:2379] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 7: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:48.758764 31351 catalog_manager.cc:5495] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 1 to 3, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 3 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:24:48.759749 31706 raft_consensus.cc:1226] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5. Ops: [2.2-4.3]
I20260805 22:24:48.751422 2394 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:48.761879 2394 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:48.762986 2378 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:48.763625 31594 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:48.766276 2293 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:48.767333 2293 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:48.742859 2452 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.741240 1952 consensus_queue.cc:1046] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.028s
I20260805 22:24:48.769629 1952 raft_consensus.cc:3032] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:48.769973 1952 raft_consensus.cc:729] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:48.770284 31706 raft_consensus.cc:1226] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 5. Current term is 6. Ops: [5.3-5.3]
I20260805 22:24:48.746197 2254 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:48.738263 31600 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:48.788482 31351 catalog_manager.cc:5495] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 1 to 3. New cstate: current_term: 3 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:48.756227 2437 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:48.791402 31351 catalog_manager.cc:5495] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 2 to 6, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 6 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:48.799433 31351 catalog_manager.cc:5495] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 1 to 2, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 2 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:48.784657 2452 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:48.806149 31351 catalog_manager.cc:5495] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 1 to 3, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 3 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
W20260805 22:24:48.808684 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 62290.5us
I20260805 22:24:48.757560 2193 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:48.798413 2437 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:48.817662 31554 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:48.816644 1981 raft_consensus.cc:2726] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:48.792230 2254 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.829558 2394 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:48.830161 2193 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:24:48.827064 2383 raft_consensus.cc:2726] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:48.831144 32585 raft_consensus.cc:1261] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260805 22:24:48.838320 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 109496us
I20260805 22:24:48.846069 2383 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:48.846733 2383 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:48.848794 2383 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:48.808902 1952 consensus_queue.cc:260] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.850685 2394 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:48.851096 31597 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:24:48.833361 2293 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:48.853639 2453 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:48.854211 32529 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:48.854364 2464 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:48.854992 2464 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.855737 32529 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:48.857342 2365 raft_consensus.cc:2781] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:24:48.854206 2453 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:48.858146 2133 raft_consensus.cc:2781] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:24:48.858662 2133 raft_consensus.cc:686] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:48.859793 2453 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:48.861070 31444 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:48.852541 31554 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:48.859541 2133 consensus_queue.cc:237] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.857159 2464 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:48.835631 2254 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:48.868240 2188 raft_consensus.cc:979] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf: : Instructing follower 516cb07ce3064ccbbca35e912d62816c to start an election
I20260805 22:24:48.868850 2188 raft_consensus.cc:1067] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Signalling peer 516cb07ce3064ccbbca35e912d62816c to start an election
I20260805 22:24:48.870282 2445 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:48.870878 2445 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.872795 32581 raft_consensus.cc:1261] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 7. (term mismatch)
I20260805 22:24:48.873149 2445 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:48.873181 32581 pending_rounds.cc:76] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf: Aborting all ops after (but not including) 6
I20260805 22:24:48.873891 2462 consensus_queue.cc:1035] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260805 22:24:48.874349 32581 pending_rounds.cc:97] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260805 22:24:48.874657 32581 raft_consensus.cc:2866] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:24:48.878971 2339 consensus_queue.cc:1035] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260805 22:24:48.879685 2365 raft_consensus.cc:686] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:48.880925 1952 raft_consensus.cc:3037] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:48.880635 2365 consensus_queue.cc:237] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.886626 31601 raft_consensus.cc:1261] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260805 22:24:48.924757 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 72532us
I20260805 22:24:48.891824 2413 raft_consensus.cc:2726] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:48.948561 31554 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:48.950569 31706 raft_consensus.cc:1261] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260805 22:24:48.924415 2464 consensus_queue.cc:1035] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.028s
I20260805 22:24:48.917950 2276 consensus_queue.cc:1035] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.036s
I20260805 22:24:48.940107 1981 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:48.963953 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:48.965504 32587 raft_consensus.cc:3032] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:48.965781 32587 raft_consensus.cc:729] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:48.968070 31554 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:48.898392 2437 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.969789 31554 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:48.971503 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:48.972173 31706 raft_consensus.cc:2454] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:48.969681 32587 consensus_queue.cc:260] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.970101 31597 raft_consensus.cc:1261] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260805 22:24:48.964254 2413 raft_consensus.cc:2726] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260805 22:24:48.975929 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 83196us
I20260805 22:24:48.977030 32587 raft_consensus.cc:3037] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:48.970649 1754 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:48.980152 31706 raft_consensus.cc:1226] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:48.980772 31554 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:48.986546 2413 raft_consensus.cc:2726] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:24:48.971148 2324 consensus_queue.cc:1046] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.014s
I20260805 22:24:48.987353 2369 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:48.987893 2369 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:48.990087 32529 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:48.990830 1989 raft_consensus.cc:2781] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:48.991174 1989 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:48.991281 31667 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:48.991549 1989 raft_consensus.cc:3037] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:48.992961 2470 consensus_queue.cc:1046] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:24:48.993964 2470 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:48.994735 2369 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:48.994920 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:48.995133 2293 raft_consensus.cc:3032] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:48.995574 2293 raft_consensus.cc:729] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:48.995652 32594 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:48.996994 31447 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:48.974877 2387 raft_consensus.cc:2726] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:48.987823 2324 raft_consensus.cc:3032] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:24:48.999382 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:49.005151 2224 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:49.009333 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:48.981659 2465 consensus_queue.cc:1035] T fe370453251f4300a8d90c397021595c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.043s
I20260805 22:24:49.009972 2174 consensus_queue.cc:1035] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
I20260805 22:24:49.011545 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:49.001823 2369 raft_consensus.cc:2781] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260805 22:24:49.057704 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 66500us
I20260805 22:24:49.059018 31667 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:49.000391 2453 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:49.064039 31667 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:49.011441 1981 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:49.008291 2324 raft_consensus.cc:729] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:49.070051 31667 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:49.074692 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 93970.4us
I20260805 22:24:49.079456 32587 raft_consensus.cc:2454] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:49.059712 2369 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:49.070660 1981 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:49.066401 2394 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:49.114689 31668 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:49.016507 2224 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:49.121714 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:49.123481 31603 raft_consensus.cc:2379] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
W20260805 22:24:49.124260 2474 log.cc:924] Time spent T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
W20260805 22:24:49.122092 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 52127.8us
I20260805 22:24:49.063390 2453 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:49.132051 2453 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:49.061076 2293 consensus_queue.cc:260] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:49.133508 2293 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:49.138962 1989 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:49.141213 2324 consensus_queue.cc:260] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:49.142625 2324 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:49.145939 32579 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:49.146277 1981 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:49.146708 2224 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:49.147320 32586 raft_consensus.cc:2454] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:49.148586 1989 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:49.149326 1981 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:49.154893 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:49.147025 2254 raft_consensus.cc:2726] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:24:49.114022 2193 raft_consensus.cc:2726] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260805 22:24:49.160665 2467 log.cc:924] Time spent T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.186s user 0.000s sys 0.001s
I20260805 22:24:49.149708 2437 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:49.080700 2466 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:49.083822 2369 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:49.169524 31667 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:24:49.164142 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 81242.8us
I20260805 22:24:49.170621 2464 raft_consensus.cc:2781] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:49.170945 31667 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:49.174405 31668 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:49.175300 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:49.180809 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:49.182370 31704 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
W20260805 22:24:49.183717 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 108329us
I20260805 22:24:49.184500 31554 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:49.185988 2413 raft_consensus.cc:2741] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election decision vote started in defunct term 5: won
I20260805 22:24:49.190588 2453 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
W20260805 22:24:49.191334 2485 log.cc:924] Time spent T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
I20260805 22:24:49.191113 2453 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:49.177487 31594 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:49.195076 31704 raft_consensus.cc:1261] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 3 index: 3. (index mismatch)
I20260805 22:24:49.195518 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:49.196491 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:49.198727 2078 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:24:49.199246 2078 raft_consensus.cc:686] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:49.200034 31715 raft_consensus.cc:2454] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:49.200495 1981 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:49.201037 1981 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:49.202003 31713 raft_consensus.cc:1226] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6. Ops: []
I20260805 22:24:49.200070 2078 consensus_queue.cc:237] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:49.203310 1981 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:49.204669 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 69876us
I20260805 22:24:49.208279 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:49.208999 31603 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:49.208344 31444 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:49.214757 31594 raft_consensus.cc:1261] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260805 22:24:49.217516 31444 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
W20260805 22:24:49.219223 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 44899.8us
I20260805 22:24:49.220602 31668 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:49.193229 2453 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:49.163975 2394 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:49.225968 2387 consensus_queue.cc:1046] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:49.227012 2394 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:49.206622 2254 raft_consensus.cc:2781] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:49.217051 2193 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:49.262979 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:49.205365 2466 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:49.242223 2300 consensus_queue.cc:1035] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.020s
I20260805 22:24:49.227556 2394 raft_consensus.cc:3032] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:49.265129 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:49.220402 2437 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:24:49.272450 31708 raft_consensus.cc:2379] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:49.238373 32529 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:49.243057 2453 raft_consensus.cc:2781] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:49.224073 2452 raft_consensus.cc:2781] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:49.223443 2313 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:49.237017 2254 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:49.275149 32529 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:49.276497 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:49.280833 1754 raft_consensus.cc:2726] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:24:49.270675 2394 raft_consensus.cc:729] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:49.276531 2365 raft_consensus.cc:2781] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:49.277362 2453 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:49.267226 2193 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:49.417011 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:24:49.382450 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 164590us
I20260805 22:24:49.411834 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:49.300400 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:24:49.362761 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 76381us
I20260805 22:24:49.414139 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:49.426702 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:49.410972 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:49.426220 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:49.385510 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:49.438902 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:49.380295 2466 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:49.444731 2466 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:49.447084 2466 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:49.277107 2452 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:49.448019 2452 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:49.279218 2313 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:49.452699 2313 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:49.411454 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:49.453876 31713 raft_consensus.cc:3037] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:49.435863 31711 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:49.415627 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:49.462862 31707 raft_consensus.cc:2379] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:49.436998 31603 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
W20260805 22:24:49.355861 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 54937.3us
I20260805 22:24:49.302919 31594 raft_consensus.cc:1226] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:49.462740 31554 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:49.414506 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:49.468623 31554 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:49.471369 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:49.472714 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:49.473037 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:49.473464 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:49.473948 1754 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:49.474509 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:49.475060 31493 raft_consensus.cc:3037] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:49.476286 31596 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:56384
I20260805 22:24:49.476859 31596 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:49.477170 31596 raft_consensus.cc:3037] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:49.477312 31704 raft_consensus.cc:2379] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 6: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:49.470724 2228 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:49.478866 2228 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:49.483117 31603 raft_consensus.cc:1261] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260805 22:24:49.481427 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:49.497361 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:49.475143 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:49.289613 2254 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:49.511741 31493 raft_consensus.cc:2454] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:49.418334 2470 raft_consensus.cc:2726] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:24:49.515843 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:49.374083 2462 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:49.463299 2453 raft_consensus.cc:3037] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:49.461805 2293 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:49.536320 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:49.541864 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:49.524971 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:49.292608 2174 raft_consensus.cc:2781] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:49.290061 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:49.526335 31608 raft_consensus.cc:2379] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 3: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:49.302626 2464 raft_consensus.cc:686] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:49.476156 2378 raft_consensus.cc:2726] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:24:49.293370 2365 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:49.527606 31596 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:49.536567 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:49.548560 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:49.547544 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:49.552595 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:49.553164 31609 raft_consensus.cc:2379] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 4: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:49.554230 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:49.574995 31603 raft_consensus.cc:2379] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 6: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:49.564898 31713 raft_consensus.cc:2454] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:49.553115 31594 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
W20260805 22:24:49.581862 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 97251.4us
I20260805 22:24:49.496909 2369 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:49.547190 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:49.543987 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:49.587534 31596 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:49.597468 31713 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
W20260805 22:24:49.599315 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 312081us
I20260805 22:24:49.567811 1998 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:49.605722 31706 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260805 22:24:49.576321 2365 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:49.569929 2437 raft_consensus.cc:686] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 6 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:49.480470 2394 consensus_queue.cc:260] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:49.581168 2078 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:49.614303 31604 raft_consensus.cc:1226] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:49.632071 2300 consensus_queue.cc:1035] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:49.644634 31554 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:49.632885 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:49.645965 31554 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:49.648320 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:49.633414 31444 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:49.633610 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:49.576752 2466 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:49.552444 2293 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:49.655787 31713 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
W20260805 22:24:49.648478 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 66726.3us
I20260805 22:24:49.652074 2324 raft_consensus.cc:2726] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:49.659955 2466 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:49.660385 2464 consensus_queue.cc:237] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:49.652591 2078 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:49.662147 2293 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:49.691824 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:49.611933 31597 raft_consensus.cc:2454] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:49.662254 2437 consensus_queue.cc:237] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:49.663102 31494 raft_consensus.cc:2354] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5.
I20260805 22:24:49.672847 31667 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:49.576166 2462 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:49.695634 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 62600.3us
I20260805 22:24:49.700069 31501 raft_consensus.cc:3037] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:49.652405 2470 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:49.673961 2193 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:49.728386 32529 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:49.738868 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 62726.4us
I20260805 22:24:49.680694 31708 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
I20260805 22:24:49.740890 31444 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:49.741412 2465 consensus_queue.cc:1046] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:24:49.742233 31444 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:49.729862 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 83953.1us
I20260805 22:24:49.743012 2465 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:49.743088 2498 raft_consensus.cc:3032] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:24:49.746084 2498 raft_consensus.cc:729] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:49.747020 2498 consensus_queue.cc:260] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:49.748332 2498 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:49.748865 1981 raft_consensus.cc:2726] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:49.749792 2494 consensus_queue.cc:1035] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:24:49.751340 2494 consensus_queue.cc:1046] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20260805 22:24:49.753731 2459 consensus_queue.cc:1046] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:24:49.755677 2413 raft_consensus.cc:3032] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:49.756086 2413 raft_consensus.cc:729] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:49.756910 2413 consensus_queue.cc:260] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:49.762465 2494 consensus_queue.cc:1046] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:49.767468 2174 raft_consensus.cc:686] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:49.768735 2174 consensus_queue.cc:237] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:49.776062 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:49.776959 32591 raft_consensus.cc:2396] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
W20260805 22:24:49.779062 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 84337.4us
I20260805 22:24:49.779860 31667 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:49.644389 1998 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:49.815555 31667 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:49.657635 2479 consensus_queue.cc:1035] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.067s
I20260805 22:24:49.706729 2293 raft_consensus.cc:2726] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260805 22:24:49.873490 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 145176us
I20260805 22:24:49.877357 32529 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:49.706348 2497 raft_consensus.cc:2726] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:49.789017 2228 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:49.888820 31492 raft_consensus.cc:1261] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260805 22:24:49.898061 2531 raft_consensus.cc:2726] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:49.876237 2465 raft_consensus.cc:2726] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:49.902972 2394 raft_consensus.cc:3037] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:24:49.903571 2495 log.cc:924] Time spent T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0: Append to log took a long time: real 0.234s user 0.000s sys 0.001s
W20260805 22:24:49.920831 2496 log.cc:924] Time spent T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.223s user 0.000s sys 0.000s
I20260805 22:24:49.839401 2193 consensus_queue.cc:1035] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.089s
W20260805 22:24:49.922760 2499 log.cc:924] Time spent T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.082s user 0.000s sys 0.000s
I20260805 22:24:49.926968 2466 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:49.928247 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:49.891853 2413 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:49.776249 2313 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:49.881099 2387 raft_consensus.cc:2726] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260805 22:24:49.974318 2491 log.cc:924] Time spent T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.314s user 0.006s sys 0.001s
I20260805 22:24:49.777109 2470 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:49.977533 2470 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:49.878415 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:49.947805 2339 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:49.964022 1998 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:49.891546 2293 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:49.893990 2188 raft_consensus.cc:2726] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:24:49.990067 2365 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:49.992717 2365 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:49.970175 2313 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:49.905761 2492 raft_consensus.cc:3032] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:24:50.011602 2492 raft_consensus.cc:729] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:50.012574 2492 consensus_queue.cc:260] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:49.890884 2267 consensus_queue.cc:1035] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.161s
I20260805 22:24:49.971508 2445 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:50.014272 2445 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:50.016258 2445 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:49.972661 2462 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:49.973737 2078 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:50.023077 2453 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:50.009285 2339 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:50.026355 2516 log.cc:924] Time spent T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.079s user 0.000s sys 0.000s
I20260805 22:24:50.044934 2339 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:50.048270 2254 consensus_meta.cc:287] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.514s user 0.006s sys 0.002s
I20260805 22:24:50.048842 2254 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:50.050451 31709 raft_consensus.cc:2379] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:50.052088 2254 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:50.009614 32586 raft_consensus.cc:3032] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:50.124485 32586 raft_consensus.cc:729] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:50.125626 32586 consensus_queue.cc:260] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:50.126833 32586 raft_consensus.cc:3037] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:24:50.028501 2452 consensus_meta.cc:287] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.580s user 0.011s sys 0.000s
I20260805 22:24:50.129725 2452 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:50.132282 2452 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:50.074875 32583 consensus_meta.cc:287] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.601s user 0.005s sys 0.005s
I20260805 22:24:50.133085 32583 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:50.133901 2390 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:50.135094 2390 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:50.135442 2390 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:50.077304 2453 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:50.108592 31501 raft_consensus.cc:2454] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:50.067935 2369 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:50.164716 2390 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:50.353145 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:50.300798 31499 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.507s user 0.001s sys 0.008s
I20260805 22:24:50.414844 31704 raft_consensus.cc:2379] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:50.176172 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:50.198309 31447 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:50.175482 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260805 22:24:50.189953 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 436978us
W20260805 22:24:50.310364 2459 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.005s sys 0.012s
W20260805 22:24:50.432786 2459 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.678s user 0.005s sys 0.014s
W20260805 22:24:50.191097 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 345304us
W20260805 22:24:50.294214 31554 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.003s sys 0.011s
W20260805 22:24:50.436611 31554 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.677s user 0.003s sys 0.016s
I20260805 22:24:50.437533 2465 raft_consensus.cc:3032] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:50.437973 2465 raft_consensus.cc:729] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:50.438843 2465 consensus_queue.cc:260] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:50.439697 2492 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:50.440104 2465 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:50.328610 31706 raft_consensus.cc:1226] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4. Ops: []
W20260805 22:24:50.266113 31495 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.541s user 0.016s sys 0.028s
I20260805 22:24:50.318387 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:50.443554 31709 raft_consensus.cc:2354] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5.
W20260805 22:24:50.267870 31601 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.533s user 0.004s sys 0.040s
I20260805 22:24:50.444130 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:50.318317 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:50.171002 32586 raft_consensus.cc:2454] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:50.350842 31715 raft_consensus.cc:1226] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 5. Current term is 6. Ops: []
I20260805 22:24:50.293910 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:50.418812 2300 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:50.447482 2494 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:50.447839 2300 raft_consensus.cc:686] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:50.448050 2494 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:50.448743 2300 consensus_queue.cc:237] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:50.449993 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:50.450579 2494 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:50.437206 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 678091us
I20260805 22:24:50.183616 2064 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:50.451887 2064 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:50.452800 32591 raft_consensus.cc:1261] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260805 22:24:50.453155 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:50.203533 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:50.454367 2550 consensus_queue.cc:1035] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:24:50.454726 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:24:50.295480 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 415719us
I20260805 22:24:50.249876 2369 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:50.456075 2369 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:50.173872 2538 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:50.457264 2538 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:50.458088 2369 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:50.459462 2538 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:50.176085 2515 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:50.246714 2535 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:50.460284 2535 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:50.246551 2536 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:50.246894 2545 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:50.461987 2545 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:50.432119 2495 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.517s user 0.000s sys 0.012s
I20260805 22:24:50.325820 32585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57372 (request call id 616) took 1097 ms. Trace:
I20260805 22:24:50.310214 2464 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:50.323558 2506 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:50.462991 2553 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260805 22:24:50.463321 32585 rpcz_store.cc:276] 0805 22:24:49.228829 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:49.262441 (+ 33612us) service_pool.cc:227] Handling call
0805 22:24:50.325589 (+1063148us) spinlock_profiling.cc:243] Waited 754 ms on lock 0x7b54005d52d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:50.325732 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1734306176}
W20260805 22:24:50.432626 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 234391us
I20260805 22:24:50.202838 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:50.194950 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:50.464639 32594 raft_consensus.cc:2454] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:50.266273 31490 raft_consensus.cc:1226] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 3. Current term is 4. Ops: []
W20260805 22:24:50.462940 2495 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.558s user 0.000s sys 0.012s
I20260805 22:24:50.466109 32579 raft_consensus.cc:2354] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4.
I20260805 22:24:50.463847 2464 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:50.463510 2506 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:50.451862 31554 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:50.467882 2464 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:50.457717 31501 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260805 22:24:50.294301 2539 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:50.468565 2506 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:50.468890 2539 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:50.469609 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:50.470727 2545 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:50.470969 2539 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:50.462517 2535 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:50.468369 2534 raft_consensus.cc:2726] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260805 22:24:50.454437 2064 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:50.473004 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:50.473402 31554 leader_election.cc:398] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5.
I20260805 22:24:50.473757 31554 leader_election.cc:401] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260805 22:24:50.467728 2553 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:24:50.463114 2515 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:50.476658 31556 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:50.477943 2515 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:50.473769 32530 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.572s user 0.001s sys 0.010s
W20260805 22:24:50.478746 32530 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.704s user 0.001s sys 0.010s
I20260805 22:24:50.479139 32581 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:50.176523 2293 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:50.480300 31447 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:50.481510 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:50.450628 31594 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:50.497536 31708 raft_consensus.cc:1261] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260805 22:24:50.498687 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:50.499547 31667 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:50.504912 31667 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:50.560978 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:50.562901 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:50.563505 31705 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
W20260805 22:24:50.564263 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 59426.6us
I20260805 22:24:50.565083 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:50.565231 31667 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:50.565850 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:50.567139 31667 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:50.567565 31711 raft_consensus.cc:2396] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for term 7 because replica has last-logged OpId of term: 4 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 3.
I20260805 22:24:50.566560 31706 raft_consensus.cc:2379] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:50.568528 31667 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:50.564975 2254 raft_consensus.cc:2726] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:24:50.495132 2383 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:50.495110 2464 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:50.498431 2515 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:50.502151 2545 raft_consensus.cc:2781] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:50.571346 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:50.593601 31667 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:50.595649 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:50.596264 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:50.596529 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:50.594508 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:50.593339 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:50.566612 2494 raft_consensus.cc:2781] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:50.598856 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:50.562579 2550 raft_consensus.cc:2781] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:50.624020 31594 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:24:50.626650 31602 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260805 22:24:50.626864 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:50.627560 31501 raft_consensus.cc:2396] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260805 22:24:50.627439 31556 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:50.618745 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:50.619776 31500 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:24:50.629632 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:50.635743 31554 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:50.637099 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:50.637524 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:50.637826 31594 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:50.638353 31500 raft_consensus.cc:2379] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:50.629451 31667 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:50.639746 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:50.640332 31711 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:50.640448 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:50.473466 2536 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:50.640072 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:50.651350 31715 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:50.648741 31609 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260805 22:24:50.590315 2452 raft_consensus.cc:2726] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:50.648967 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:50.590067 2228 raft_consensus.cc:2726] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:24:50.573665 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:50.657502 31608 raft_consensus.cc:1261] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:24:50.573000 2546 raft_consensus.cc:2726] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:50.658906 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:50.502368 2293 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:50.618949 2545 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:50.632838 2534 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:50.632608 2254 raft_consensus.cc:2726] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:50.681844 31594 raft_consensus.cc:1261] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260805 22:24:50.682359 31704 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:50.626789 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:50.685353 31494 raft_consensus.cc:1799] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Handling vote request from an unknown peer feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:50.696287 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:50.646375 2550 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:50.688467 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:50.696187 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:50.698837 31494 raft_consensus.cc:2396] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 3 because replica has last-logged OpId of term: 2 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 4.
I20260805 22:24:50.712786 31502 raft_consensus.cc:1261] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260805 22:24:50.598990 2464 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:50.630906 2515 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:50.660898 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:50.699283 2534 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:50.719142 2293 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:50.693894 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:50.699919 2525 raft_consensus.cc:3037] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:50.715299 2550 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:50.724062 31597 raft_consensus.cc:2396] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 7 because replica has last-logged OpId of term: 5 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 2.
I20260805 22:24:50.725131 31705 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:24:50.648478 2224 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.050s
I20260805 22:24:50.691653 2545 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:24:50.730275 2558 log.cc:924] Time spent T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.126s user 0.001s sys 0.003s
I20260805 22:24:50.694608 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:50.739866 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:50.879444 2515 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:50.903714 31602 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:24:50.886222 31490 raft_consensus.cc:1261] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:24:50.744246 31711 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:50.812446 31604 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:24:50.905758 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 278391us
I20260805 22:24:50.788877 31667 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
W20260805 22:24:50.835309 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 199736us
W20260805 22:24:50.911146 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 122335us
I20260805 22:24:50.911283 31554 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:50.912668 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:50.818396 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 330151us
I20260805 22:24:50.913297 31494 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
W20260805 22:24:50.880270 2560 log.cc:924] Time spent T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.187s user 0.001s sys 0.000s
I20260805 22:24:50.714960 2390 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:50.912155 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:50.812076 31714 raft_consensus.cc:2379] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:50.924098 31602 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:24:50.925693 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:50.928421 31447 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:50.878548 2534 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:50.933499 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:50.942207 31667 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:50.947855 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:50.948724 31602 raft_consensus.cc:2396] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for term 8 because replica has last-logged OpId of term: 6 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260805 22:24:50.949026 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:50.815192 32585 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:50.783078 31486 raft_consensus.cc:2454] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:50.951035 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:50.954104 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:50.952517 31604 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:50.917135 2293 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:50.912487 32579 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:50.955015 31490 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:24:50.922866 2466 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:50.972028 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:50.982679 2466 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:50.983295 31705 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:50.984122 2383 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5.
I20260805 22:24:50.984694 2466 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:50.943293 2064 raft_consensus.cc:2781] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:50.984992 2383 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:50.985544 2064 raft_consensus.cc:686] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
W20260805 22:24:50.987459 2492 consensus_meta.cc:287] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.547s user 0.006s sys 0.006s
I20260805 22:24:50.978118 2078 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:24:50.977689 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:50.954466 2534 raft_consensus.cc:2726] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:24:50.978595 2464 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:50.943077 2494 raft_consensus.cc:686] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:50.972338 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:50.997958 31609 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:50.927945 2254 consensus_queue.cc:1035] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.071s
I20260805 22:24:51.001364 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:50.951566 2536 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:50.981891 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:51.002929 31495 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:51.010591 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:51.011262 31715 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:51.012609 32529 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:51.005897 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:51.004745 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:51.017787 31667 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:50.981578 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:51.022918 31600 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:51.023056 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:51.005259 2078 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:51.024181 2078 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:51.029531 2464 raft_consensus.cc:2726] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:51.021895 31667 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:51.009488 2492 raft_consensus.cc:2726] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:51.037990 31667 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:51.042423 2563 log.cc:924] Time spent T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.117s user 0.000s sys 0.001s
I20260805 22:24:51.031288 2548 raft_consensus.cc:2781] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:51.042805 2539 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:51.004041 2383 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:51.061014 2561 log.cc:924] Time spent T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.224s user 0.001s sys 0.001s
I20260805 22:24:51.043246 2466 raft_consensus.cc:2726] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260805 22:24:51.097760 31423 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica f8e9f4c756da4203917c464e2b179994 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:24:51.101428 2383 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:51.102147 2494 consensus_queue.cc:237] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:51.103214 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 89a7aa19faed455fb2600ae6851aee0e, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:51.103574 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 89a7aa19faed455fb2600ae6851aee0e, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.061954 2548 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:51.107244 2548 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:51.085507 2064 consensus_queue.cc:237] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:51.111765 2525 raft_consensus.cc:2726] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:51.114241 31447 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:51.116034 31711 raft_consensus.cc:1226] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:51.116782 31667 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:51.121136 31667 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:51.128839 2574 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
W20260805 22:24:51.136140 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet eda619941ce64f788d0b7a364eed0bdb, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:51.136523 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet eda619941ce64f788d0b7a364eed0bdb, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.136240 31667 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:51.084182 2452 consensus_queue.cc:1035] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.045s
I20260805 22:24:51.126788 2538 raft_consensus.cc:2781] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:51.119426 2383 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:51.075806 2539 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:51.125725 2228 raft_consensus.cc:2781] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:51.143285 2276 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:51.119573 2582 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:51.130663 2254 raft_consensus.cc:2726] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:24:51.154235 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:51.119761 2466 consensus_queue.cc:1035] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
W20260805 22:24:51.155059 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 243846us
I20260805 22:24:51.155997 31707 raft_consensus.cc:2379] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
W20260805 22:24:51.156425 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 0ff925b5d86b41c19d809f76e57ab062, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:51.156752 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 0ff925b5d86b41c19d809f76e57ab062, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.140722 2538 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:51.157491 31554 leader_election.cc:398] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Vote denied by peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607) with higher term. Message: Invalid argument: T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5.
W20260805 22:24:51.158437 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 152400us
I20260805 22:24:51.147571 2228 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:51.168697 31667 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:51.138468 2574 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.174192 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:51.174896 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:51.175709 31601 raft_consensus.cc:2379] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 4: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:51.171082 31444 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
W20260805 22:24:51.157836 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 247329us
I20260805 22:24:51.152452 2506 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:51.174858 31494 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:51.178161 2506 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.178593 2078 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.179819 31705 raft_consensus.cc:2379] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 8: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:51.180388 2506 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:51.180764 2078 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:51.181562 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:51.182403 31704 raft_consensus.cc:2379] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:51.187768 2545 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:51.189483 31499 raft_consensus.cc:2454] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:24:51.190758 2545 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:51.173352 2228 raft_consensus.cc:3037] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:51.196671 2539 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:51.157615 2582 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:51.154484 2492 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:51.154062 2276 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.149163 2383 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.164503 2538 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:51.206218 31668 leader_election.cc:398] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Vote denied by peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801) with higher term. Message: Invalid argument: T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4.
I20260805 22:24:51.233929 31668 leader_election.cc:401] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260805 22:24:51.206657 2470 raft_consensus.cc:2781] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:51.208865 32529 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
W20260805 22:24:51.222708 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 10de517cc0a24f20b5885091e99e252d, TS 516cb07ce3064ccbbca35e912d62816c -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:51.236173 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 10de517cc0a24f20b5885091e99e252d, TS 516cb07ce3064ccbbca35e912d62816c -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:51.215432 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 44434.3us
W20260805 22:24:51.206126 2575 log.cc:924] Time spent T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.131s user 0.000s sys 0.001s
W20260805 22:24:51.222930 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 60194.4us
I20260805 22:24:51.237150 31608 raft_consensus.cc:2454] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:51.249214 31444 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:51.259289 31667 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
W20260805 22:24:51.260466 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 54442.6us
I20260805 22:24:51.231294 2506 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election won for term 7
W20260805 22:24:51.244992 2577 log.cc:924] Time spent T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.072s user 0.000s sys 0.000s
I20260805 22:24:51.276489 31444 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:51.288079 32586 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
mode: GRACEFUL
new_leader_uuid: "f8e9f4c756da4203917c464e2b179994"
from {username='slave'} at 127.0.0.1:60186
I20260805 22:24:51.288462 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:51.289113 31486 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:51.185145 2550 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.245900 1989 raft_consensus.cc:2781] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:51.298324 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:51.299005 31594 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:51.248211 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:51.263377 2539 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:51.267138 2545 raft_consensus.cc:2726] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:51.234951 2492 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.267441 2254 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:51.303712 31667 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:51.293821 2470 raft_consensus.cc:686] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:51.303069 2452 raft_consensus.cc:2726] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801) with higher term. Message: Invalid argument: T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4.
W20260805 22:24:51.322808 2573 log.cc:924] Time spent T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
W20260805 22:24:51.362309 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 58704.6us
I20260805 22:24:51.360121 31594 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:24:51.288022 2531 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
W20260805 22:24:51.362748 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 183617us
I20260805 22:24:51.392391 31556 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:51.295759 2584 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:51.393905 31667 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:51.303938 2293 raft_consensus.cc:2781] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:51.396186 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:51.397233 32581 raft_consensus.cc:2379] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 4: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:51.398321 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:51.303224 2078 raft_consensus.cc:2781] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:51.399536 2078 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
W20260805 22:24:51.317873 2575 log.cc:924] Time spent T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.057s user 0.000s sys 0.001s
W20260805 22:24:51.388089 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 117002us
I20260805 22:24:51.305882 1989 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260805 22:24:51.399940 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 95831.7us
I20260805 22:24:51.336176 2536 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:51.323379 2464 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:51.408084 31444 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
W20260805 22:24:51.402796 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 130461us
I20260805 22:24:51.425654 31444 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:51.424809 31351 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34388 (request call id 279) took 7465 ms. Trace:
I20260805 22:24:51.399969 2078 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:51.407183 1989 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:51.429520 31351 rpcz_store.cc:276] 0805 22:24:43.942648 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:43.942829 (+ 181us) service_pool.cc:227] Handling call
0805 22:24:51.407984 (+7465155us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0805 22:24:48.994984 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0805 22:24:48.995608 (+ 624us) write_op.cc:432] Acquiring schema lock in shared mode
0805 22:24:48.995676 (+ 68us) write_op.cc:435] Acquired schema lock
0805 22:24:48.995688 (+ 12us) tablet.cc:660] Decoding operations
0805 22:24:49.139144 (+143456us) write_op.cc:620] Acquiring the partition lock for write op
0805 22:24:49.139248 (+ 104us) write_op.cc:641] Partition lock acquired for write op
0805 22:24:49.139275 (+ 27us) tablet.cc:683] Acquiring locks for 17 operations
0805 22:24:49.140523 (+ 1248us) tablet.cc:699] Row locks acquired
0805 22:24:49.140565 (+ 42us) write_op.cc:260] PREPARE: finished
0805 22:24:49.140726 (+ 161us) write_op.cc:270] Start()
0805 22:24:49.140877 (+ 151us) write_op.cc:276] Timestamp: P: 1785968689140661 usec, L: 0
0805 22:24:49.140900 (+ 23us) op_driver.cc:331] REPLICATION: starting
0805 22:24:49.227269 (+ 86369us) log.cc:842] Serialized 6768 byte log entry
0805 22:24:50.566535 (+1339266us) op_driver.cc:444] REPLICATION: finished
0805 22:24:51.086481 (+519946us) write_op.cc:301] APPLY: starting
0805 22:24:51.086566 (+ 85us) tablet.cc:1365] starting BulkCheckPresence
0805 22:24:51.086848 (+ 282us) tablet.cc:1368] finished BulkCheckPresence
0805 22:24:51.093909 (+ 7061us) tablet.cc:1370] starting ApplyRowOperation cycle
0805 22:24:51.098319 (+ 4410us) tablet.cc:1381] finished ApplyRowOperation cycle
0805 22:24:51.098609 (+ 290us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=17
0805 22:24:51.098654 (+ 45us) write_op.cc:312] APPLY: finished
0805 22:24:51.098994 (+ 340us) log.cc:842] Serialized 119 byte log entry
0805 22:24:51.099159 (+ 165us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:51.099324 (+ 165us) write_op.cc:454] Released schema lock
0805 22:24:51.099373 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0805 22:24:51.110313 (+ 10940us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:51.110344 (+ 31us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":155031,"reported_tablets":33,"child_traces":[["op",{"apply.queue_time_us":493082,"apply.run_cpu_time_us":4392,"apply.run_wall_time_us":24312,"num_ops":17,"prepare.queue_time_us":93811,"prepare.run_cpu_time_us":57153,"prepare.run_wall_time_us":511867,"raft.queue_time_us":576257,"raft.run_cpu_time_us":18010,"raft.run_wall_time_us":469868,"replication_time_us":1361543,"thread_start_us":1146196,"threads_started":4,"wal-append.queue_time_us":416983}]]}
I20260805 22:24:51.362025 2539 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.460999 31444 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:51.389008 2254 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.450186 31356 catalog_manager.cc:5495] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:51.427871 2506 raft_consensus.cc:686] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
W20260805 22:24:51.493432 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 68898.5us
I20260805 22:24:51.536655 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:51.537563 31708 raft_consensus.cc:2379] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 4: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:51.250523 2569 consensus_queue.cc:1035] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.209s
I20260805 22:24:51.542191 31554 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:51.442348 2609 raft_consensus.cc:2781] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:51.544998 31356 catalog_manager.cc:5495] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 2 to 3, leader changed from 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 3 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:51.467041 2494 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:51.457806 2607 raft_consensus.cc:2781] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:51.459875 2604 raft_consensus.cc:2781] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:51.394352 2584 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:51.588395 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 326590us
I20260805 22:24:51.596510 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:51.391690 2531 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:51.581211 2575 log.cc:924] Time spent T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20260805 22:24:51.604842 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:51.632230 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:51.634820 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:51.635977 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:51.637562 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:51.450294 31698 tablet_service.cc:1514] Processing DeleteTablet for tablet ab66683a9e5e46c2b09be4afdee2a08c with delete_type TABLET_DATA_TOMBSTONED (TS feb5e7ab4b1541beaf2674c3c5500385 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:56246
I20260805 22:24:51.635010 31704 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:51.591643 2193 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election won for term 8
I20260805 22:24:51.645601 31495 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:51.646903 31709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 2184) took 1148 ms. Trace:
I20260805 22:24:51.591291 2607 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:51.596706 2276 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:51.592340 31356 catalog_manager.cc:5495] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:51.597091 2604 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:51.596378 2574 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:51.649562 31709 rpcz_store.cc:276] 0805 22:24:50.496829 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:50.496974 (+ 145us) service_pool.cc:227] Handling call
0805 22:24:50.502887 (+ 5913us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:50.597650 (+ 94763us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:50.597683 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:50.645591 (+ 47908us) log.cc:842] Serialized 26 byte log entry
0805 22:24:50.929765 (+284174us) raft_consensus.cc:1631] Marking committed up to 3
0805 22:24:51.241479 (+311714us) log.cc:842] Serialized 14 byte log entry
0805 22:24:51.644994 (+403515us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:51.645146 (+ 152us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:51.645194 (+ 48us) raft_consensus.cc:1671] finished
0805 22:24:51.645205 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:51.645339 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":137023,"prepare.queue_time_us":118717,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"raft.queue_time_us":218361,"raft.run_cpu_time_us":315,"raft.run_wall_time_us":45730,"thread_start_us":858725,"threads_started":3,"wal-append.queue_time_us":428379}
I20260805 22:24:51.600273 2582 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:51.630728 2609 raft_consensus.cc:686] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:51.610216 2611 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:51.646540 2520 raft_consensus.cc:2726] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260805 22:24:51.667537 2548 consensus_meta.cc:287] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.560s user 0.008s sys 0.003s
I20260805 22:24:51.720980 2590 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:51.722972 2625 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:51.723692 2625 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.611354 2536 consensus_queue.cc:1035] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.082s
I20260805 22:24:51.596800 2494 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.725662 31492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40740 (request call id 1166) took 1105 ms. Trace:
I20260805 22:24:51.653789 2550 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:24:51.727444 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:51.667250 2607 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:51.682126 2605 raft_consensus.cc:2726] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:24:51.683447 2616 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:51.686878 2619 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:51.678234 2604 raft_consensus.cc:3037] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
W20260805 22:24:51.724318 2594 log.cc:924] Time spent T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
I20260805 22:24:51.732462 31492 rpcz_store.cc:276] 0805 22:24:50.596514 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:50.596623 (+ 109us) service_pool.cc:227] Handling call
0805 22:24:50.641586 (+ 44963us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:50.648378 (+ 6792us) raft_consensus.cc:1520] Early marking committed up to index 6
0805 22:24:50.660032 (+ 11654us) log.cc:842] Serialized 14 byte log entry
0805 22:24:50.723456 (+ 63424us) spinlock_profiling.cc:243] Waited 7.95 ms on lock 0x7b180043efa0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e140638 00007f272e143b11 00007f272e143ac9 00007f272e143a59 00007f272e143a21 00007f272e142d1d 00007f27373a4204 00007f27373a4039 00007f272e122dc6
0805 22:24:50.922221 (+198765us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:50.986402 (+ 64181us) spinlock_profiling.cc:243] Waited 2.54 ms on lock 0x7b2c0003d328. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272fa4be11 00007f272fa5b442 00007f27310a3c11 00007f273106858a 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772
0805 22:24:50.986727 (+ 325us) log.cc:842] Serialized 26 byte log entry
0805 22:24:50.986839 (+ 112us) raft_consensus.cc:1631] Marking committed up to 7
0805 22:24:50.987047 (+ 208us) log.cc:842] Serialized 14 byte log entry
0805 22:24:50.987222 (+ 175us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:50.987285 (+ 63us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:51.701472 (+714187us) raft_consensus.cc:1671] finished
0805 22:24:51.701506 (+ 34us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:51.701669 (+ 163us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24712,"prepare.queue_time_us":27253,"prepare.run_cpu_time_us":66,"prepare.run_wall_time_us":66,"raft.queue_time_us":8134,"raft.run_cpu_time_us":281,"raft.run_wall_time_us":27854,"spinlock_wait_cycles":24124544,"thread_start_us":222275,"threads_started":1,"wal-append.queue_time_us":353908}
I20260805 22:24:51.694089 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:51.761593 31356 catalog_manager.cc:5495] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 3, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 3 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:51.700646 2492 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:51.753829 2581 log.cc:924] Time spent T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.367s user 0.000s sys 0.001s
I20260805 22:24:51.705778 2569 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:51.770714 2625 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:51.644340 2470 consensus_queue.cc:237] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.704041 2582 raft_consensus.cc:2781] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:51.803181 31356 catalog_manager.cc:5495] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260805 22:24:51.779861 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 245238us
I20260805 22:24:51.786785 31486 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:24:51.787546 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:51.811213 31495 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:51.753739 2224 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:51.759706 2193 raft_consensus.cc:686] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
W20260805 22:24:51.818251 2596 log.cc:924] Time spent T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.170s user 0.000s sys 0.000s
I20260805 22:24:51.819094 2193 consensus_queue.cc:237] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.831880 2293 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:51.832476 2293 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:51.840662 2464 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:51.842564 2531 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:51.692788 2611 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:51.732892 2548 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:51.804850 31356 catalog_manager.cc:5495] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 1 to 3, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 3 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:51.817718 2224 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:51.844556 2618 log.cc:924] Time spent T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260805 22:24:51.774080 2619 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.731468 2506 consensus_queue.cc:237] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.766150 2616 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.855192 2584 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:51.798764 2586 log.cc:924] Time spent T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.373s user 0.001s sys 0.001s
I20260805 22:24:51.796303 2582 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:51.841272 2464 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
W20260805 22:24:51.860739 2600 log.cc:924] Time spent T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.359s user 0.001s sys 0.000s
I20260805 22:24:51.896934 31356 catalog_manager.cc:5495] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:24:51.875871 2383 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:51.877585 2254 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:51.802791 2609 consensus_queue.cc:237] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:51.857236 2539 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:51.882484 2494 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:51.884387 2224 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:51.882503 2638 tablet_replica.cc:330] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385: stopping tablet replica
I20260805 22:24:51.904913 2638 raft_consensus.cc:2229] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260805 22:24:51.905576 32586 raft_consensus.cc:595] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Received request to transfer leadership to TS f8e9f4c756da4203917c464e2b179994
I20260805 22:24:51.905736 2638 raft_consensus.cc:2258] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260805 22:24:51.885785 2611 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:51.889058 31706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1760) took 1382 ms. Trace:
I20260805 22:24:51.906075 32586 raft_consensus.cc:603] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:24:51.906701 2548 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:51.906947 2611 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:51.907516 2611 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.908438 2619 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:51.909366 2611 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:51.912953 2638 ts_tablet_manager.cc:1905] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20260805 22:24:51.913765 2078 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:51.915699 2228 consensus_meta.cc:287] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.670s user 0.009s sys 0.000s
I20260805 22:24:51.916009 2078 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:24:51.916692 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 59428.4us
I20260805 22:24:51.906332 31706 rpcz_store.cc:276] 0805 22:24:50.506600 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:50.590069 (+ 83469us) service_pool.cc:227] Handling call
0805 22:24:50.630073 (+ 40004us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:50.683205 (+ 53132us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:50.683239 (+ 34us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:50.713433 (+ 30194us) log.cc:842] Serialized 26 byte log entry
0805 22:24:51.047086 (+333653us) spinlock_profiling.cc:243] Waited 17.6 ms on lock 0x7b4800020358. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2735f0f7f1 00007f2735f068e7 00007f2735f0677a 00007f2735ffa95f 00007f2735ffa627 00007f273600eb2c 00007f273600ead8 00007f273600ea58 00007f273600ea13 00007f273600dcbc 00007f2734f8fbe6
0805 22:24:51.315315 (+268229us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:24:51.315360 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:51.315442 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:51.888777 (+573335us) raft_consensus.cc:1671] finished
0805 22:24:51.888797 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:51.889001 (+ 204us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5409,"raft.queue_time_us":33634,"raft.run_cpu_time_us":3725,"raft.run_wall_time_us":357571,"spinlock_wait_cycles":40503680,"thread_start_us":469553,"threads_started":1,"wal-append.queue_time_us":1154920}
I20260805 22:24:51.889809 2616 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:51.794164 2569 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.919471 2604 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:51.920460 2569 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:51.863933 2445 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:51.921183 2445 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:51.923218 2445 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:51.923353 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:51.923820 31594 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:51.924924 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:51.959244 31601 raft_consensus.cc:2379] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 4: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:51.958750 2607 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.960618 2616 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:51.961133 2616 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.963179 2616 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:51.963353 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:51.963807 2445 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:51.964051 2569 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:51.916105 2228 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.964599 2569 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:51.965027 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:51.965186 31708 raft_consensus.cc:1226] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:51.966548 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:51.967365 2445 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:51.969475 2445 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:51.957525 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:51.974522 31494 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:51.975329 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:51.976599 2607 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:51.977067 1989 consensus_meta.cc:287] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.510s user 0.005s sys 0.005s
I20260805 22:24:51.980443 32581 raft_consensus.cc:1261] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260805 22:24:51.980614 2637 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:51.981168 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:51.949050 31594 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:24:51.986110 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:51.993558 31604 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:51.985611 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:51.990012 31667 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:51.995311 31502 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:51.996260 31495 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:24:51.998379 31667 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:51.873620 2582 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:51.975598 2228 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:52.006182 2078 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:52.014523 2624 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:51.993127 2616 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:51.987071 2569 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:52.022689 2538 consensus_meta.cc:287] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.776s user 0.011s sys 0.000s
I20260805 22:24:52.022750 31667 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:51.936738 2604 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.005947 2520 raft_consensus.cc:2726] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:24:52.031134 31667 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:52.005769 2619 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
W20260805 22:24:52.032577 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 497998us
I20260805 22:24:52.035605 31444 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:52.016918 2293 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:51.994788 1989 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.030763 2538 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.049757 31667 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:52.049269 31444 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
W20260805 22:24:52.052848 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 510710us
I20260805 22:24:52.048435 32529 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:52.073918 31554 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:52.030558 2464 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.036902 2300 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:52.077013 31444 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:52.078486 31554 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:52.079648 2640 raft_consensus.cc:2781] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:52.089887 31554 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:52.031046 2616 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.052578 2064 raft_consensus.cc:2726] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:24:52.047742 2254 raft_consensus.cc:2781] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:24:52.080453 2584 raft_consensus.cc:2726] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:52.098541 31444 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:52.099128 2506 raft_consensus.cc:2726] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:24:52.093905 2494 raft_consensus.cc:2726] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:52.032157 2078 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.094252 2466 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:52.092046 2445 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:52.101651 2609 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:52.070503 2604 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:52.045303 2619 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.087814 2452 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:52.080188 2646 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:52.048719 2633 consensus_queue.cc:1035] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.083s
I20260805 22:24:51.996213 2637 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.128271 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:52.143733 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.094635 2539 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:52.129925 31444 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:52.029764 2624 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.132814 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:52.148377 31594 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:52.102011 2625 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:52.126438 32585 raft_consensus.cc:2354] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5.
I20260805 22:24:52.141981 31668 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:52.124539 2224 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:52.150229 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.124795 2535 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:52.154234 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:52.154842 31708 raft_consensus.cc:2362] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8. Re-sending same reply.
I20260805 22:24:52.131381 2520 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:52.155864 31704 raft_consensus.cc:1261] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:24:52.095686 2300 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.155967 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.159659 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:52.160269 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.152424 31713 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:52.162245 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.132764 2254 raft_consensus.cc:686] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:52.164354 31709 raft_consensus.cc:2379] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:52.164634 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.170194 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.165295 31608 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:52.170799 31706 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:52.173956 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.140129 2174 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:52.184643 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:52.180083 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.185599 31501 raft_consensus.cc:2454] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:52.184057 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:52.149173 2539 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:52.139971 2602 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
W20260805 22:24:52.170317 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 121951us
I20260805 22:24:52.187656 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.186960 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.174400 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.174669 31705 raft_consensus.cc:2379] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 6: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:52.178362 31594 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
I20260805 22:24:52.190163 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.189342 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.190974 31492 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:52.191315 31601 raft_consensus.cc:1226] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6. Ops: []
I20260805 22:24:52.174810 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.192538 31713 raft_consensus.cc:2379] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 6: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:52.161491 2617 raft_consensus.cc:2781] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:52.201205 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.202913 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.205307 31490 raft_consensus.cc:1261] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:24:52.203512 31499 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:52.136804 2445 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.183866 2470 raft_consensus.cc:2726] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:52.201930 31594 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:52.193832 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.195921 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.197845 2640 raft_consensus.cc:686] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:52.214318 2644 raft_consensus.cc:2781] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:52.214774 2628 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:52.215199 2626 raft_consensus.cc:2726] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:24:52.215845 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.235177 31492 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:52.141721 2452 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.141551 2604 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.139439 2606 consensus_queue.cc:1046] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.072s
I20260805 22:24:52.189245 2539 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:52.151803 2609 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.225857 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.224195 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.157050 2625 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.223652 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.203045 2661 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:52.197479 2631 raft_consensus.cc:2781] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:52.226778 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.193223 31554 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:52.230376 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.199529 2617 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:52.148835 2646 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.200619 2506 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:52.232229 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.200826 2607 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:52.236768 2538 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.174470 2520 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.237004 32594 raft_consensus.cc:1226] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6. Ops: []
I20260805 22:24:52.295763 2637 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.162382 2224 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.237298 2293 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.242877 2619 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:52.243646 2616 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.239044 31708 raft_consensus.cc:2362] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4. Re-sending same reply.
I20260805 22:24:52.226975 31714 raft_consensus.cc:2379] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:52.227945 31706 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:52.247943 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.247871 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:52.248952 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:52.342787 32586 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:52.308753 2078 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.257313 31499 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:52.255059 31502 raft_consensus.cc:2379] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:52.261327 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.261530 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.197351 2623 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:52.354300 2625 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:52.354791 2623 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.256561 31705 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:52.356815 2623 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:52.287742 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 105755us
I20260805 22:24:52.234565 2644 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:52.357412 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.357802 2644 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:52.358124 31706 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:52.238432 2612 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:52.358945 2612 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.361156 2612 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.365315 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.366631 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.366947 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:52.368125 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:52.368712 31594 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:52.349489 32586 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:52.371277 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.366008 31500 raft_consensus.cc:2454] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:52.238854 2452 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:52.238503 2445 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:52.238229 2624 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.240859 2582 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.375548 2644 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.328233 2383 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:52.252897 1989 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:52.316001 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 143522us
I20260805 22:24:52.377005 2383 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.254422 2464 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:52.288149 2617 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:52.374667 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.349238 2538 consensus_queue.cc:1046] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260805 22:24:52.378433 31596 raft_consensus.cc:2454] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:52.195259 2602 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.322818 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.237183 2628 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.190300 2174 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.296252 2569 raft_consensus.cc:2781] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:52.291917 2631 raft_consensus.cc:686] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:52.231906 2254 consensus_queue.cc:237] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.342893 31711 raft_consensus.cc:3037] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:52.250834 2539 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.341414 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:52.236675 2640 consensus_queue.cc:237] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.351737 2664 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:52.352396 2656 consensus_queue.cc:1046] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.057s
I20260805 22:24:52.353834 2224 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:52.365496 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.376471 2582 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.233807 2536 consensus_queue.cc:1035] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.059s
I20260805 22:24:52.376257 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.303309 2506 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.306217 2607 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:52.289630 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 96469.4us
I20260805 22:24:52.317368 31444 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:52.285424 2661 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.474786 2609 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.516847 31709 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:52.468127 31667 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:52.517619 2661 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:24:52.517781 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 49742.7us
I20260805 22:24:52.518241 2661 raft_consensus.cc:2781] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:52.518638 2661 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:52.518987 2661 raft_consensus.cc:3037] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:52.519208 2539 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:52.519837 31604 raft_consensus.cc:1226] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 6. Current term is 7. Ops: []
I20260805 22:24:52.514914 32585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57382 (request call id 638) took 1305 ms. Trace:
I20260805 22:24:52.520727 32585 rpcz_store.cc:276] 0805 22:24:51.209189 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:51.221617 (+ 12428us) service_pool.cc:227] Handling call
0805 22:24:52.514532 (+1292915us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b54004f51d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:52.514789 (+ 257us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1758859008}
I20260805 22:24:52.521400 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.356683 2646 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.522542 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.460186 2617 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.523996 31600 raft_consensus.cc:2379] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:52.524125 31704 raft_consensus.cc:2354] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6.
I20260805 22:24:52.524376 2633 raft_consensus.cc:2781] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:52.525156 2633 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:52.525055 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.525662 2633 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:52.526842 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.446593 31705 raft_consensus.cc:1261] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260805 22:24:52.417187 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:52.477605 2383 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.531206 31554 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:52.385286 31707 raft_consensus.cc:2379] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:52.383533 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:52.532383 31602 raft_consensus.cc:2454] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:52.552347 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.423477 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:52.553795 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.444839 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:52.431586 2644 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.523325 2607 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.554749 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:52.554381 31447 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:52.534334 2602 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.534945 2224 raft_consensus.cc:3032] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:52.534973 2664 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.555722 2224 raft_consensus.cc:729] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:52.554802 31706 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:52.425904 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.377789 31668 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:52.538367 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 338845us
I20260805 22:24:52.479952 2604 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.473958 2520 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:24:52.385171 2647 log.cc:924] Time spent T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.253s user 0.000s sys 0.001s
I20260805 22:24:52.527117 2617 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:52.536564 2569 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:52.541772 2174 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:52.411854 2300 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:52.542855 31604 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:52.544118 2506 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:52.544804 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:52.546962 32594 raft_consensus.cc:2454] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:52.553233 31494 raft_consensus.cc:2396] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260805 22:24:52.553079 2646 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:52.554169 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.554702 2383 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:52.555501 31486 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:52.557250 2224 consensus_queue.cc:260] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.618023 2224 raft_consensus.cc:3037] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:52.536214 31603 raft_consensus.cc:1261] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260805 22:24:52.560542 2664 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:52.560575 2630 raft_consensus.cc:3032] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:52.561455 2607 raft_consensus.cc:2781] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:52.515048 2638 ts_tablet_manager.cc:1918] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20260805 22:24:52.597012 2569 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:52.524107 2611 raft_consensus.cc:3032] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:24:52.576027 31594 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:24:52.540534 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
W20260805 22:24:52.572326 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 194615us
W20260805 22:24:52.521521 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 204250us
I20260805 22:24:52.578780 31667 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:52.573845 31711 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:52.587224 31596 raft_consensus.cc:1261] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260805 22:24:52.594308 31554 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:52.556310 2631 consensus_queue.cc:237] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.594326 2604 raft_consensus.cc:2781] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:52.573896 2520 raft_consensus.cc:2726] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:52.603225 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:52.605120 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:52.610633 31600 raft_consensus.cc:2396] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 7 because replica has last-logged OpId of term: 5 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260805 22:24:52.625244 31486 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:24:52.562163 31447 leader_election.cc:398] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Vote denied by peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801) with higher term. Message: Invalid argument: T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5.
W20260805 22:24:52.683058 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 104352us
I20260805 22:24:52.621074 2630 raft_consensus.cc:729] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:52.684172 2604 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:52.684635 2604 raft_consensus.cc:3037] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:52.631106 2383 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.684916 2630 consensus_queue.cc:260] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.684650 2531 raft_consensus.cc:2726] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:52.686117 31447 leader_election.cc:401] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260805 22:24:52.686296 2630 raft_consensus.cc:3037] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:24:52.686606 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 124583us
I20260805 22:24:52.667017 2638 log.cc:1196] T ab66683a9e5e46c2b09be4afdee2a08c P feb5e7ab4b1541beaf2674c3c5500385: Deleting WAL directory at /tmp/dist-test-taskFRO9g9/test-tmp/auto_leader_rebalancer-test.0.LeaderRebalancerTest.AddTserver.1785968555326349-29783-0/minicluster-data/ts-2-root/wals/ab66683a9e5e46c2b09be4afdee2a08c
I20260805 22:24:52.687778 31447 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:52.678056 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:52.690023 2300 raft_consensus.cc:2781] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:52.690408 2300 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:52.690738 2300 raft_consensus.cc:3037] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:52.692133 2604 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.671125 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:52.683651 32587 raft_consensus.cc:3037] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 5
I20260805 22:24:52.668995 32591 raft_consensus.cc:1261] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:24:52.669471 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:52.643934 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:52.694438 31609 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:52.628435 32529 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:52.695271 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 101043us
I20260805 22:24:52.698621 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:52.615283 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:52.667507 31706 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:52.700618 31444 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
W20260805 22:24:52.703303 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 74957.7us
I20260805 22:24:52.702934 31554 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:52.705255 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:52.705905 31554 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:52.688570 2644 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:52.646586 2611 raft_consensus.cc:729] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:52.710942 2539 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:52.717228 2611 consensus_queue.cc:260] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.717763 2539 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:52.718148 2539 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:52.718431 2611 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:52.711269 31554 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:52.721704 2228 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:52.722252 2228 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.724288 2228 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.727701 2637 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:52.728211 2637 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.729708 32587 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:52.710621 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:52.712018 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:52.722297 31667 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:52.686637 2383 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:52.667742 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:52.688591 2638 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:56246 (request call id 819) took 1277 ms. Trace:
I20260805 22:24:52.715448 31609 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:24:52.682225 2628 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:52.646883 2607 raft_consensus.cc:686] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:52.744692 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:52.674602 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:52.702363 2602 raft_consensus.cc:2781] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:52.758960 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:52.749194 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:52.759965 31711 raft_consensus.cc:2354] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5.
I20260805 22:24:52.748793 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.731686 2637 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.758285 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:52.762233 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.645561 2664 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.088s
I20260805 22:24:52.631314 2646 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.637053 2633 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.794344 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:52.763432 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.845898 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.846688 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:52.847263 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.794751 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:52.734184 2640 raft_consensus.cc:2741] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260805 22:24:52.766938 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.708211 2582 consensus_queue.cc:1035] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.085s
I20260805 22:24:52.851819 31444 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:52.856242 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.768586 31667 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:52.743510 2545 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:52.847420 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:24:52.807473 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 53377.8us
I20260805 22:24:52.744491 2616 raft_consensus.cc:2741] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260805 22:24:52.768075 2602 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260805 22:24:52.858590 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 90102.2us
I20260805 22:24:52.859009 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.767608 2383 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:52.859841 31715 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:52.852910 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.865353 31705 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:24:52.841113 2604 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:52.793594 2628 raft_consensus.cc:2781] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:52.862401 31667 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:52.862665 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.870792 32529 leader_election.cc:398] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5.
I20260805 22:24:52.874311 32529 leader_election.cc:401] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260805 22:24:52.876354 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.877786 32529 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:52.879942 31596 raft_consensus.cc:1261] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260805 22:24:52.881098 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.882908 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.710578 2682 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.154s
I20260805 22:24:52.894563 32529 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:52.760754 2300 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.859684 2545 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:52.796916 2661 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.871120 2538 consensus_queue.cc:1046] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.040s
I20260805 22:24:52.864542 2602 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:52.865010 2383 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:52.863379 2535 raft_consensus.cc:2726] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:52.859824 2531 raft_consensus.cc:2726] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260805 22:24:52.919113 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.735023 2638 rpcz_store.cc:276] 0805 22:24:51.437455 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:51.449960 (+ 12505us) service_pool.cc:227] Handling call
0805 22:24:51.882072 (+432112us) ts_tablet_manager.cc:1145] Deleting tablet ab66683a9e5e46c2b09be4afdee2a08c
0805 22:24:51.882191 (+ 119us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0805 22:24:52.240771 (+358580us) tablet_metadata.cc:641] Metadata flushed
0805 22:24:52.514766 (+273995us) tablet_metadata.cc:641] Metadata flushed
0805 22:24:52.688497 (+173731us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":231778,"thread_start_us":258779,"threads_started":1}
I20260805 22:24:52.931322 31502 raft_consensus.cc:2379] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:52.888497 2637 raft_consensus.cc:2781] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:52.918190 31667 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:52.886210 2648 raft_consensus.cc:2726] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5.
I20260805 22:24:52.908061 2445 raft_consensus.cc:2726] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:52.797842 2607 consensus_queue.cc:237] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:52.877334 2628 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:52.914424 2545 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:52.914281 2470 consensus_queue.cc:1035] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.047s
W20260805 22:24:52.960769 2689 log.cc:924] Time spent T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.110s user 0.000s sys 0.000s
I20260805 22:24:52.961395 2300 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.919042 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
W20260805 22:24:52.964366 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 46295.5us
I20260805 22:24:52.930548 2383 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:52.968923 31715 raft_consensus.cc:2454] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
W20260805 22:24:52.969120 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 120871us
I20260805 22:24:52.924500 2617 consensus_queue.cc:1035] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.049s
I20260805 22:24:52.978489 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.963771 2637 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:52.979493 31596 raft_consensus.cc:2379] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 7: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:52.981011 31715 raft_consensus.cc:1226] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 6. Current term is 7. Ops: []
I20260805 22:24:52.970347 2531 raft_consensus.cc:2781] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:24:52.982002 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.984169 31596 raft_consensus.cc:1261] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260805 22:24:52.984715 31499 raft_consensus.cc:1261] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260805 22:24:52.986366 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.986627 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.987277 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.987648 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:52.987884 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:52.988961 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.989629 31600 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:52.990163 2633 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:52.991063 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:52.991295 2661 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:52.991850 31713 raft_consensus.cc:2379] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:52.994377 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.994669 31715 raft_consensus.cc:1226] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:52.995087 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.995326 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.995872 31711 raft_consensus.cc:2454] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:24:52.996783 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.990391 31597 raft_consensus.cc:2379] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 7: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:52.998168 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:52.998018 2539 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:52.998778 31706 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:24:52.999276 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:52.999656 31554 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:53.000084 2539 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:52.999686 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:53.000597 2582 raft_consensus.cc:2781] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:53.001098 2582 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:52.989784 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:53.001549 2582 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:53.002735 2644 raft_consensus.cc:2726] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801) with higher term. Message: Invalid argument: T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5.
I20260805 22:24:52.974624 2174 raft_consensus.cc:2726] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:53.004112 2646 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:52.973315 2628 raft_consensus.cc:3037] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:53.009653 31492 raft_consensus.cc:2454] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:24:53.011703 31501 raft_consensus.cc:1261] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260805 22:24:53.012969 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:53.013868 31609 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:53.015566 2582 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:53.017091 2582 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:53.018055 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:53.020794 31554 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:53.023214 31554 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:53.023334 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:53.025153 31495 raft_consensus.cc:3037] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:53.025509 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:53.026319 31705 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:53.026638 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:53.027235 31713 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:53.025604 31554 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:53.034480 31554 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:53.034480 31667 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:53.035558 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 165057us
I20260805 22:24:53.039896 31554 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:52.981962 2637 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:53.034217 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:53.041568 31667 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:53.042028 31707 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:53.042384 31554 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:53.048669 31495 raft_consensus.cc:2454] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:53.050832 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:53.021262 2531 raft_consensus.cc:686] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:53.058724 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:53.058804 2666 consensus_queue.cc:1035] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:53.059060 2531 consensus_queue.cc:237] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:53.060004 2550 consensus_queue.cc:1035] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:53.038713 2539 raft_consensus.cc:2781] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:53.064292 2539 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:53.064694 2539 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:53.034090 2582 raft_consensus.cc:2726] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:53.066485 2582 raft_consensus.cc:2781] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:53.066984 2582 raft_consensus.cc:686] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:53.067855 2582 consensus_queue.cc:237] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:53.055130 2254 raft_consensus.cc:2726] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:24:53.056479 2628 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:53.041132 2604 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:53.070078 2604 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:53.070492 2604 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:53.042543 2646 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:53.072314 2646 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:53.044917 2606 raft_consensus.cc:2781] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:53.073675 2606 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:53.074072 2606 raft_consensus.cc:3037] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:53.074469 2646 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:53.080202 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:53.080595 31667 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:53.080966 31495 raft_consensus.cc:2379] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
W20260805 22:24:53.082113 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 187628us
I20260805 22:24:53.086658 32529 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:53.086825 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:53.087695 31500 raft_consensus.cc:1226] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260805 22:24:53.090831 31608 raft_consensus.cc:2379] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:53.091854 32529 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:53.097632 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:53.076858 2631 raft_consensus.cc:2726] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:24:53.095722 2656 raft_consensus.cc:2781] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:53.100879 31667 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:53.099594 2656 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:53.103196 2656 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:53.103916 32579 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
mode: GRACEFUL
new_leader_uuid: "f8e9f4c756da4203917c464e2b179994"
from {username='slave'} at 127.0.0.1:60186
I20260805 22:24:53.097965 2617 consensus_queue.cc:1035] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260805 22:24:53.110862 31444 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:53.089612 2646 raft_consensus.cc:2781] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:53.118856 31444 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:53.107486 2470 raft_consensus.cc:2726] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:53.122150 2646 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:53.124122 31707 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:24:53.104307 31600 raft_consensus.cc:1261] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260805 22:24:53.102321 32529 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.194:41897 timed out after 1.838s (SENT) [suppressed 13 similar messages]
I20260805 22:24:53.141235 31711 raft_consensus.cc:1226] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 6. Current term is 7. Ops: [5.3-6.4]
I20260805 22:24:53.129134 32579 raft_consensus.cc:595] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Received request to transfer leadership to TS f8e9f4c756da4203917c464e2b179994
I20260805 22:24:53.133066 31444 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:53.100237 2445 raft_consensus.cc:2781] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:53.135816 2646 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:53.099370 2706 consensus_queue.cc:1046] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.047s
I20260805 22:24:53.131425 2708 raft_consensus.cc:2726] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:53.133905 2506 raft_consensus.cc:2726] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:53.169189 32579 raft_consensus.cc:603] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:24:53.191524 31444 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
W20260805 22:24:53.188881 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 152578us
W20260805 22:24:53.189656 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 147393us
I20260805 22:24:53.171645 2535 raft_consensus.cc:2726] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:53.186852 2445 raft_consensus.cc:686] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:53.169849 2545 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:53.220817 31444 leader_election.cc:398] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6.
I20260805 22:24:53.221181 31444 leader_election.cc:401] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
W20260805 22:24:53.222594 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 120295us
I20260805 22:24:53.223572 2506 consensus_queue.cc:1035] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:24:53.225236 2506 raft_consensus.cc:2726] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6.
I20260805 22:24:53.223104 31444 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:53.230446 2712 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:53.230962 2712 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:53.231345 2712 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:53.244035 31609 raft_consensus.cc:1261] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:53.242699 31444 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:53.203668 2617 consensus_queue.cc:1035] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.051s
I20260805 22:24:53.247799 31486 raft_consensus.cc:1261] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
W20260805 22:24:53.296443 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 55992.7us
I20260805 22:24:53.303097 2731 consensus_queue.cc:1035] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:24:53.287691 32579 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:60186
I20260805 22:24:53.304652 32579 raft_consensus.cc:595] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:53.305035 32579 raft_consensus.cc:603] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:24:53.300308 2633 raft_consensus.cc:2781] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:53.306218 2633 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:53.306635 2633 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:53.310355 2520 consensus_queue.cc:1035] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:24:53.313623 32587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:60186
I20260805 22:24:53.314247 32587 raft_consensus.cc:595] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:53.315541 32583 raft_consensus.cc:1261] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 7. (term mismatch)
I20260805 22:24:53.316025 32583 pending_rounds.cc:76] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf: Aborting all ops after (but not including) 6
I20260805 22:24:53.316282 32583 pending_rounds.cc:97] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf: Aborting uncommitted NO_OP operation due to leader change: 4.7
I20260805 22:24:53.316536 32583 raft_consensus.cc:2866] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:24:53.317696 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:53.318832 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:53.307607 2602 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:53.258131 2699 log.cc:924] Time spent T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.067s user 0.000s sys 0.000s
I20260805 22:24:53.218292 2716 raft_consensus.cc:2726] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:53.217612 2643 raft_consensus.cc:2741] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election decision vote started in defunct term 4: won
W20260805 22:24:53.281720 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 88002.6us
I20260805 22:24:53.325973 31668 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:53.330153 2637 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:53.331497 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:53.332466 2276 raft_consensus.cc:3032] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:24:53.333706 2545 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:53.334842 2656 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:53.337622 32594 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:60186
I20260805 22:24:53.338029 2539 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:53.320130 2569 consensus_meta.cc:287] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.626s user 0.006s sys 0.005s
I20260805 22:24:53.249094 2452 consensus_queue.cc:1035] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.095s
I20260805 22:24:53.345769 31444 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:53.345917 2604 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:53.355016 2383 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:53.368321 2646 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:53.378876 2633 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:53.349803 2606 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:53.261855 2445 consensus_queue.cc:237] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:53.968837 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 566686us
W20260805 22:24:53.969461 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 643522us
I20260805 22:24:53.763774 2607 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:53.975327 2383 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:53.755676 2520 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:53.735129 2616 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
W20260805 22:24:53.540800 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 334790us
I20260805 22:24:53.987623 31556 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:53.813588 2602 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:53.974213 31554 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:53.431277 2193 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260805 22:24:53.982764 31706 raft_consensus.cc:1226] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 5. Current term is 6. Ops: []
I20260805 22:24:53.970491 2276 raft_consensus.cc:729] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:53.971376 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:53.974162 32594 raft_consensus.cc:595] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:53.735877 2465 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:53.763751 2661 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:53.997267 31554 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:24:54.002969 31608 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.780s user 0.004s sys 0.064s
I20260805 22:24:53.985255 32583 raft_consensus.cc:1226] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6. Ops: []
I20260805 22:24:53.975497 2607 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:54.009502 31608 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.786s user 0.004s sys 0.065s
W20260805 22:24:54.014003 31495 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.803s user 0.023s sys 0.044s
I20260805 22:24:53.976722 2569 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:54.003257 1989 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.637s user 0.008s sys 0.003s
W20260805 22:24:54.014416 31495 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.803s user 0.023s sys 0.044s
I20260805 22:24:54.015097 31554 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:54.001828 2383 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:54.001366 2728 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:54.003916 2706 raft_consensus.cc:2781] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:53.993666 2739 raft_consensus.cc:2726] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:54.031562 31554 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:54.039659 31594 raft_consensus.cc:1226] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5. Ops: []
W20260805 22:24:54.027590 1989 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.671s user 0.009s sys 0.003s
I20260805 22:24:54.008014 32581 raft_consensus.cc:3037] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:53.991703 2520 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.013154 2636 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260805 22:24:53.991704 2616 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.033468 2716 raft_consensus.cc:2726] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260805 22:24:54.025969 2628 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:54.025172 2602 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:24:54.035930 2706 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
W20260805 22:24:54.021335 2531 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.669s user 0.012s sys 0.036s
W20260805 22:24:54.124562 2531 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.784s user 0.012s sys 0.037s
I20260805 22:24:54.013638 2661 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.082933 2739 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:54.084539 2735 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:54.041147 2276 consensus_queue.cc:260] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.089768 2637 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:54.090446 2539 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:54.091674 2607 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:54.087919 2520 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:54.094273 2606 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:54.108166 2604 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:54.108712 2740 log.cc:924] Time spent T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.139s user 0.000s sys 0.001s
I20260805 22:24:54.033646 2383 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.038625 2728 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:54.053216 2729 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:24:54.114823 31594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1191) took 1134 ms. Trace:
I20260805 22:24:54.118364 2569 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:54.011566 2465 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:54.127604 2545 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:54.137697 31594 rpcz_store.cc:276] 0805 22:24:52.980411 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:52.980551 (+ 140us) service_pool.cc:227] Handling call
0805 22:24:52.980832 (+ 281us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:54.041510 (+1060678us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:54.114505 (+ 72995us) spinlock_profiling.cc:243] Waited 1.06 s on lock 0x7b540003f7d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:54.114718 (+ 213us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2434590464}
I20260805 22:24:54.126008 31596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1197) took 1030 ms. Trace:
I20260805 22:24:54.150624 31596 rpcz_store.cc:276] 0805 22:24:53.096088 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:53.096215 (+ 127us) service_pool.cc:227] Handling call
0805 22:24:53.096501 (+ 286us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:53.209861 (+113360us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:53.209896 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:53.220383 (+ 10487us) log.cc:842] Serialized 26 byte log entry
0805 22:24:53.584197 (+363814us) raft_consensus.cc:1631] Marking committed up to 2
0805 22:24:53.584241 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:53.584400 (+ 159us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:54.125806 (+541406us) raft_consensus.cc:1671] finished
0805 22:24:54.125825 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:54.125949 (+ 124us) inbound_call.cc:173] Queueing success response
0805 22:24:54.132993 (+ 7044us) spinlock_profiling.cc:243] Waited 5.63 ms on lock 0x7b4800020c58. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2735f0f7f1 00007f2735f068e7 00007f2735f0677a 00007f2735ffa95f 00007f2735ffa627 00007f273600eb2c 00007f273600ead8 00007f273600ea58 00007f273600ea13 00007f273600dcbc 00007f2734f8fbe6
Metrics: {"mutex_wait_us":1453,"raft.queue_time_us":160024,"raft.run_cpu_time_us":3735,"raft.run_wall_time_us":876299,"spinlock_wait_cycles":12956544,"thread_start_us":443543,"threads_started":2,"wal-append.queue_time_us":806030}
I20260805 22:24:54.090536 2633 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:54.155857 2545 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.158077 2545 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:54.164268 2728 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:54.120759 2712 consensus_meta.cc:287] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.889s user 0.012s sys 0.002s
I20260805 22:24:54.166875 2712 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.168876 2712 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:54.123222 2602 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:54.169984 2602 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:54.131912 2276 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:54.124054 2706 raft_consensus.cc:3037] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:54.129016 2739 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.176854 2739 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:54.138841 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 163704us
I20260805 22:24:54.178445 31668 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:54.179677 31668 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:54.129863 2735 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.155473 2465 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:54.181296 31668 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:54.278957 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 97723.1us
I20260805 22:24:54.279811 31668 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:54.165088 32581 raft_consensus.cc:2454] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:54.285081 31668 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:54.286762 31668 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:54.287868 31668 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:54.150704 2664 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:54.291586 31668 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:54.291805 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 946175us
I20260805 22:24:54.173645 2383 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:54.152526 2729 raft_consensus.cc:686] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 7 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:54.192308 2665 raft_consensus.cc:2726] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:54.277765 2470 raft_consensus.cc:2726] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:54.293780 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:54.296080 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:54.304639 31594 raft_consensus.cc:2354] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6.
I20260805 22:24:54.303233 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260805 22:24:54.289816 2617 raft_consensus.cc:2726] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:54.290163 2604 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:54.291886 2631 raft_consensus.cc:2726] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:54.290088 2520 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
W20260805 22:24:54.359771 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 52444.9us
I20260805 22:24:54.365043 31444 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:54.292922 2637 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:54.363407 31609 raft_consensus.cc:1226] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:54.381488 2735 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:54.382351 2661 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:54.383068 2729 consensus_queue.cc:237] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.373829 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:54.359480 2665 raft_consensus.cc:2726] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:54.395771 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:54.399178 31495 raft_consensus.cc:2354] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 6. Current term is 7.
I20260805 22:24:54.396922 31493 raft_consensus.cc:1226] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 6. Current term is 7. Ops: []
I20260805 22:24:54.299443 2531 consensus_queue.cc:1035] T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.076s
I20260805 22:24:54.376068 31554 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:54.364370 2545 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:54.398643 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:54.374552 2631 raft_consensus.cc:2726] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:54.368944 2745 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:54.405573 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:54.374958 2437 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:54.407090 31609 raft_consensus.cc:1226] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 6. Current term is 7. Ops: []
I20260805 22:24:54.293115 2664 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:54.413722 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:54.421052 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:24:54.401885 31600 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.880s user 0.007s sys 0.008s
I20260805 22:24:54.422206 31602 raft_consensus.cc:2354] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 6. Current term is 7.
W20260805 22:24:54.422204 31600 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.911s user 0.008s sys 0.008s
I20260805 22:24:54.387382 2607 raft_consensus.cc:2726] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:54.419155 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:54.406509 31554 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:54.427465 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:54.428082 31495 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:54.369599 2604 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.416200 31608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1196) took 1320 ms. Trace:
I20260805 22:24:54.427603 2582 raft_consensus.cc:2726] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:24:54.418949 2735 raft_consensus.cc:2726] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:54.436602 31554 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:54.435660 31608 rpcz_store.cc:276] 0805 22:24:53.095772 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:53.096654 (+ 882us) service_pool.cc:227] Handling call
0805 22:24:53.114088 (+ 17434us) raft_consensus.cc:1457] Updating replica for 2 ops
0805 22:24:53.222147 (+108059us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:53.222187 (+ 40us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0805 22:24:53.222808 (+ 621us) log.cc:842] Serialized 52 byte log entry
0805 22:24:54.011661 (+788853us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:24:54.402241 (+390580us) log.cc:842] Serialized 14 byte log entry
0805 22:24:54.405308 (+ 3067us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:54.405428 (+ 120us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:54.405471 (+ 43us) raft_consensus.cc:1671] finished
0805 22:24:54.405483 (+ 12us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:54.416138 (+ 10655us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1868,"raft.queue_time_us":157849,"raft.run_cpu_time_us":464,"raft.run_wall_time_us":50228,"thread_start_us":1241988,"threads_started":3,"wal-append.queue_time_us":910015}
W20260805 22:24:54.417104 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 126210us
I20260805 22:24:54.416374 2383 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:54.438946 31554 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:54.429195 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 459252us
I20260805 22:24:54.446048 32529 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:54.449182 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:54.449326 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:54.449811 31594 raft_consensus.cc:2362] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6. Re-sending same reply.
I20260805 22:24:54.451714 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:54.456367 31602 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:54.452732 31594 raft_consensus.cc:1261] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:24:54.451294 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:54.458781 31492 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:54.430902 2616 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:54.426151 2437 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.424115 2745 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.428094 2749 consensus_queue.cc:1035] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.067s
I20260805 22:24:54.445755 2383 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.490239 2752 raft_consensus.cc:2726] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:54.497519 2437 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:54.494328 2747 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:54.498660 2747 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.499704 31444 leader_election.cc:398] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6.
I20260805 22:24:54.500006 31444 leader_election.cc:401] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260805 22:24:54.500713 2747 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:54.418228 2545 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:54.503014 2706 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.503979 2545 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:54.504123 31604 raft_consensus.cc:1226] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:54.433614 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:54.446352 2739 raft_consensus.cc:2726] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:24:54.451076 2728 consensus_queue.cc:1046] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.055s
I20260805 22:24:54.506879 2728 raft_consensus.cc:3032] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:54.507234 2728 raft_consensus.cc:729] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
W20260805 22:24:54.507845 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 68094.9us
I20260805 22:24:54.508051 2728 consensus_queue.cc:260] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:54.509044 31447 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:54.450876 2716 raft_consensus.cc:2781] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:54.509896 2716 raft_consensus.cc:686] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:54.510748 2716 consensus_queue.cc:237] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.512460 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:54.513111 32589 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:54.513464 2535 raft_consensus.cc:2781] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:54.513974 2535 raft_consensus.cc:686] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:54.451238 2746 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:54.481057 2664 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:54.480386 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:54.489113 2602 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.507682 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:54.519979 2745 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:54.497885 2383 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:54.485517 2604 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:54.520423 2602 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:54.521072 2718 raft_consensus.cc:3032] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:24:54.521539 2718 raft_consensus.cc:729] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:54.522229 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:54.522856 31596 raft_consensus.cc:2454] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:54.522428 2718 consensus_queue.cc:260] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.523226 31597 raft_consensus.cc:1226] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260805 22:24:54.523653 2745 consensus_queue.cc:1046] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 3.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:54.525619 2745 raft_consensus.cc:3032] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:54.525794 2656 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:54.526000 2745 raft_consensus.cc:729] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:54.453552 2666 consensus_queue.cc:1046] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.073s
I20260805 22:24:54.468125 31608 raft_consensus.cc:1226] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 6. Current term is 7. Ops: [6.8-6.8]
I20260805 22:24:54.470989 31495 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:54.531448 31600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1199) took 1335 ms. Trace:
I20260805 22:24:54.528445 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:54.532136 31706 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:54.533353 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:54.534041 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:54.534690 32594 raft_consensus.cc:2454] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:54.534722 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:54.535264 31447 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:54.536501 31447 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:54.539256 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:54.540663 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:54.535437 31500 raft_consensus.cc:1261] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:54.544941 31492 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:24:54.546280 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:54.550571 31444 leader_election.cc:398] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 6. Current term is 7.
I20260805 22:24:54.550876 31444 leader_election.cc:401] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260805 22:24:54.546939 31596 raft_consensus.cc:2454] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:54.554792 31596 raft_consensus.cc:1226] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260805 22:24:54.556715 31706 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:54.559309 31603 raft_consensus.cc:1226] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:54.562211 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:54.563603 31486 raft_consensus.cc:1226] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 6. Current term is 7. Ops: []
I20260805 22:24:54.564586 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:54.531854 31600 rpcz_store.cc:276] 0805 22:24:53.196257 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:53.196413 (+ 156us) service_pool.cc:227] Handling call
0805 22:24:53.196789 (+ 376us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:53.493018 (+296229us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:53.493056 (+ 38us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:53.510860 (+ 17804us) log.cc:842] Serialized 26 byte log entry
0805 22:24:54.437075 (+926215us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:24:54.437128 (+ 53us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:54.437192 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:54.531213 (+ 94021us) raft_consensus.cc:1671] finished
0805 22:24:54.531233 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:54.531395 (+ 162us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":105696,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"thread_start_us":1207677,"threads_started":2,"wal-append.queue_time_us":934690}
I20260805 22:24:54.566535 31444 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:54.565474 31554 leader_election.cc:398] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Vote denied by peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607) with higher term. Message: Invalid argument: T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 6. Current term is 7.
I20260805 22:24:54.452018 2648 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:54.595067 2648 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:54.595721 2193 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:54.526904 2745 consensus_queue.cc:260] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.596225 2193 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.596567 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:54.596737 2745 raft_consensus.cc:3037] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:54.596943 2607 raft_consensus.cc:2726] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 6. Current term is 7.
I20260805 22:24:54.597364 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:54.597971 31708 raft_consensus.cc:3037] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:54.596235 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:54.600319 31704 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:54.600539 2648 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:54.601778 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:54.602239 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:54.602856 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:54.603229 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:54.603873 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:54.604247 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:54.604444 31714 raft_consensus.cc:2454] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:54.605048 31602 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:24:54.606077 2706 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:54.568619 31706 raft_consensus.cc:1226] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 6. Current term is 7. Ops: []
I20260805 22:24:54.566032 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:54.608693 31711 raft_consensus.cc:2379] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:54.605232 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:54.609925 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:54.447247 32581 raft_consensus.cc:1261] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260805 22:24:54.451067 2617 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:54.589530 31707 raft_consensus.cc:1261] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260805 22:24:54.591647 2766 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:54.592098 2612 raft_consensus.cc:2781] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:54.558858 2437 raft_consensus.cc:2781] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:54.592972 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:54.593338 2228 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:54.593743 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:54.594009 2609 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:54.594272 2531 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:54.594556 2520 raft_consensus.cc:2726] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:24:54.615760 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:54.566700 2383 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:54.691210 31706 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:54.692241 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:54.693023 31706 raft_consensus.cc:2379] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 7: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:54.587421 31554 leader_election.cc:401] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
W20260805 22:24:54.693809 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 128513us
I20260805 22:24:54.695385 31554 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:54.696151 31554 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:54.616528 31704 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:54.698949 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:54.699553 31704 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:54.612164 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:54.702258 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:54.548213 2728 raft_consensus.cc:3037] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:54.619129 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:54.706214 31608 raft_consensus.cc:3037] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:54.620419 31600 raft_consensus.cc:1226] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:54.625797 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:54.528844 2636 raft_consensus.cc:2726] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6.
I20260805 22:24:54.612138 31708 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:54.631023 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:54.711460 31493 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:54.558279 2276 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:54.711660 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:54.712224 2276 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:54.712615 2276 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:54.712926 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:54.713696 31708 raft_consensus.cc:2379] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:54.592550 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:54.643309 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:54.651582 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:54.652058 2760 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:54.652659 2193 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:54.717116 2760 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.654058 31444 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:54.717804 31711 raft_consensus.cc:1226] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6. Ops: []
W20260805 22:24:54.718395 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 64374.1us
I20260805 22:24:54.655835 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:54.720094 2465 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:24:54.657660 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:54.658385 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:54.659053 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:54.630764 32583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57382 (request call id 658) took 1295 ms. Trace:
I20260805 22:24:54.641454 2633 consensus_queue.cc:1046] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260805 22:24:54.722267 31490 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:54.640735 2617 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:54.666224 2766 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.652664 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:54.514807 2535 consensus_queue.cc:237] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:54.617172 32579 raft_consensus.cc:2454] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:54.612887 2545 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:54.566010 2718 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:54.652082 2646 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:54.653316 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:54.753414 31604 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:54.676501 31357 catalog_manager.cc:5495] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 2 to 5, leader changed from 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 5 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260805 22:24:54.756628 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 302548us
I20260805 22:24:54.757349 32529 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:54.566234 2751 consensus_queue.cc:1046] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.101s
I20260805 22:24:54.732857 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:54.643853 2747 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:54.566252 2746 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:54.667783 2612 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:54.675972 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:24:54.733475 2750 consensus_queue.cc:1046] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.080s
I20260805 22:24:54.724866 2531 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.674564 2437 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:54.724359 2748 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:54.729265 2749 consensus_queue.cc:1046] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 3.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.082s
I20260805 22:24:54.690564 2228 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.724700 2609 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.750254 31709 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:54.729328 2383 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:54.840315 31423 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
I20260805 22:24:54.746219 31704 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
W20260805 22:24:54.798478 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 45563.1us
I20260805 22:24:54.806756 31711 raft_consensus.cc:1226] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:54.751685 31608 raft_consensus.cc:2454] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:54.875648 31554 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
W20260805 22:24:54.862506 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 105225us
I20260805 22:24:54.734211 2705 raft_consensus.cc:2726] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607) with higher term. Message: Invalid argument: T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 6. Current term is 7.
I20260805 22:24:54.814806 2716 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:54.758922 2739 raft_consensus.cc:2781] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:24:54.843281 2617 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:54.876784 32529 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:54.864082 2437 raft_consensus.cc:3037] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:54.878515 31704 raft_consensus.cc:1226] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 6. Current term is 7. Ops: []
W20260805 22:24:54.881798 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 196096us
I20260805 22:24:54.759455 2616 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:24:54.860158 2612 raft_consensus.cc:3037] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:54.874533 31715 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:54.846453 2760 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:54.892378 31554 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:54.926678 31554 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:54.876368 31713 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 8.
I20260805 22:24:54.892841 32529 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:54.857174 2602 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
W20260805 22:24:54.938186 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 212683us
I20260805 22:24:54.929716 32529 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:54.897164 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:54.752537 2193 consensus_queue.cc:1035] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.126s
I20260805 22:24:54.759265 2545 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.813191 2731 consensus_queue.cc:1046] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.078s
I20260805 22:24:54.864049 2747 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:54.882900 2757 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:54.925182 2749 raft_consensus.cc:3032] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:54.870977 2748 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.965672 2757 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:54.965672 2749 raft_consensus.cc:729] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:54.891481 2648 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:54.967314 2648 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:54.967428 2749 consensus_queue.cc:260] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.891698 2445 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:54.900137 2746 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:54.882782 2604 raft_consensus.cc:2781] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:54.969368 2604 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:54.969774 2604 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:54.901470 2664 raft_consensus.cc:2726] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:54.895063 2656 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:54.971321 2656 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:54.972168 2787 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:54.902292 2739 raft_consensus.cc:686] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 6 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:54.844453 2766 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:54.904305 31356 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47946 (request call id 185) took 9021 ms. Trace:
I20260805 22:24:52.971654 31339 catalog_manager.cc:4841] TS feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): tablet ab66683a9e5e46c2b09be4afdee2a08c (table test-workload [id=7bd7c746eae544c8b845d4350c0ba52f]) successfully deleted
I20260805 22:24:54.904764 2751 raft_consensus.cc:3032] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:54.905120 2773 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:54.906785 31357 catalog_manager.cc:5495] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 2 to 5, leader changed from 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 5 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:54.858275 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:54.907634 2750 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:54.938211 2767 raft_consensus.cc:2726] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:54.985445 31357 catalog_manager.cc:5495] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 4 to 5, leader changed from 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 5 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:54.909789 2643 raft_consensus.cc:3032] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:24:54.993680 2643 raft_consensus.cc:729] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:54.994060 2705 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:54.870878 2752 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:54.996296 2545 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:54.954062 2706 raft_consensus.cc:2726] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:54.954119 2174 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:24:54.954425 2637 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:54.905153 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:54.998256 31357 catalog_manager.cc:5495] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:54.936606 31715 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:24:54.798090 2729 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:54.939206 31704 raft_consensus.cc:3037] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:24:55.000242 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 119719us
I20260805 22:24:55.001125 31357 catalog_manager.cc:5495] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 3 to 5, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 5 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:24:54.857309 2607 consensus_queue.cc:1035] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.046s
I20260805 22:24:54.952234 31554 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:54.956557 31709 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
W20260805 22:24:54.957952 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 60993.8us
I20260805 22:24:54.723625 32583 rpcz_store.cc:276] 0805 22:24:53.335270 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:53.338098 (+ 2828us) service_pool.cc:227] Handling call
0805 22:24:53.971883 (+633785us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:54.012575 (+ 40692us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:54.630437 (+617862us) spinlock_profiling.cc:243] Waited 4.77 ms on lock 0x7b5400652054. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:54.630676 (+ 239us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":10980736}
I20260805 22:24:55.013645 31667 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:54.962504 2709 consensus_queue.cc:1046] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260805 22:24:54.972677 2787 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:54.973426 31339 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 2.00161e+06us
I20260805 22:24:54.986521 31492 raft_consensus.cc:1261] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260805 22:24:54.955175 2602 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.018612 31492 pending_rounds.cc:76] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994: Aborting all ops after (but not including) 3
I20260805 22:24:54.996734 2383 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:54.952414 2582 raft_consensus.cc:2781] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:54.968684 2749 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:55.022418 2747 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:55.023041 2739 consensus_queue.cc:237] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.990712 2773 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:55.005681 2174 raft_consensus.cc:686] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:55.027889 2174 consensus_queue.cc:237] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:55.017575 2706 raft_consensus.cc:2781] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:54.994568 2705 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.002905 2637 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.911818 2531 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:54.941459 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:54.973500 2751 raft_consensus.cc:729] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:55.008137 31704 raft_consensus.cc:2454] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
W20260805 22:24:55.012506 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 60369.4us
I20260805 22:24:55.038023 2582 raft_consensus.cc:686] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:55.038445 2751 consensus_queue.cc:260] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:55.040366 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 80759.7us
I20260805 22:24:55.040840 31704 raft_consensus.cc:1226] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:55.043717 31704 raft_consensus.cc:1261] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260805 22:24:55.044708 31704 raft_consensus.cc:1226] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:55.046250 31554 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:55.047802 31554 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:55.013842 31707 raft_consensus.cc:1261] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:24:55.014170 31711 raft_consensus.cc:1226] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 6. Current term is 7. Ops: []
I20260805 22:24:54.893128 2716 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.968665 2445 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:54.996812 2750 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:54.951377 2276 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:54.958037 2670 consensus_queue.cc:1035] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.043s
I20260805 22:24:55.016412 31667 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:55.098456 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:54.973120 31356 rpcz_store.cc:276] 0805 22:24:45.884117 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:45.886708 (+ 2591us) service_pool.cc:227] Handling call
0805 22:24:54.904230 (+9017522us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0805 22:24:52.161431 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0805 22:24:52.217612 (+ 56181us) write_op.cc:432] Acquiring schema lock in shared mode
0805 22:24:52.217650 (+ 38us) write_op.cc:435] Acquired schema lock
0805 22:24:52.217667 (+ 17us) tablet.cc:660] Decoding operations
0805 22:24:52.218528 (+ 861us) write_op.cc:620] Acquiring the partition lock for write op
0805 22:24:52.218590 (+ 62us) write_op.cc:641] Partition lock acquired for write op
0805 22:24:52.218616 (+ 26us) tablet.cc:683] Acquiring locks for 7 operations
0805 22:24:52.219158 (+ 542us) tablet.cc:699] Row locks acquired
0805 22:24:52.219177 (+ 19us) write_op.cc:260] PREPARE: finished
0805 22:24:52.219328 (+ 151us) write_op.cc:270] Start()
0805 22:24:52.219438 (+ 110us) write_op.cc:276] Timestamp: P: 1785968692219281 usec, L: 0
0805 22:24:52.219459 (+ 21us) op_driver.cc:331] REPLICATION: starting
0805 22:24:52.284349 (+ 64890us) log.cc:842] Serialized 2835 byte log entry
0805 22:24:52.632619 (+348270us) op_driver.cc:444] REPLICATION: finished
0805 22:24:52.908126 (+275507us) write_op.cc:301] APPLY: starting
0805 22:24:52.913834 (+ 5708us) tablet.cc:1365] starting BulkCheckPresence
0805 22:24:52.919073 (+ 5239us) tablet.cc:1368] finished BulkCheckPresence
0805 22:24:52.919207 (+ 134us) tablet.cc:1370] starting ApplyRowOperation cycle
0805 22:24:52.920041 (+ 834us) tablet.cc:1381] finished ApplyRowOperation cycle
0805 22:24:52.920270 (+ 229us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0805 22:24:52.920316 (+ 46us) write_op.cc:312] APPLY: finished
0805 22:24:52.920581 (+ 265us) log.cc:842] Serialized 59 byte log entry
0805 22:24:52.920753 (+ 172us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:52.920867 (+ 114us) write_op.cc:454] Released schema lock
0805 22:24:52.920911 (+ 44us) write_op.cc:341] FINISH: Updating metrics
0805 22:24:52.921227 (+ 316us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:52.921253 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":94970,"reported_tablets":21,"child_traces":[["op",{"apply.queue_time_us":240051,"apply.run_cpu_time_us":3031,"apply.run_wall_time_us":41500,"num_ops":7,"prepare.queue_time_us":225856,"prepare.run_cpu_time_us":7933,"prepare.run_wall_time_us":375277,"raft.queue_time_us":125530,"raft.run_cpu_time_us":32699,"raft.run_wall_time_us":215767,"replication_time_us":413064,"thread_start_us":611988,"threads_started":4,"wal-append.queue_time_us":213794,"wal-append.run_cpu_time_us":5948,"wal-append.run_wall_time_us":1476163}]]}
I20260805 22:24:55.018889 31492 pending_rounds.cc:97] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260805 22:24:55.099375 31608 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
I20260805 22:24:55.099654 31492 raft_consensus.cc:2866] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:24:55.030581 2706 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:55.017514 2762 raft_consensus.cc:2726] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:55.019119 31357 catalog_manager.cc:5495] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:55.038074 2705 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:55.052625 31704 raft_consensus.cc:1226] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:55.057076 31554 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:55.095575 32586 raft_consensus.cc:1226] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260805 22:24:55.097165 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:55.095351 32594 raft_consensus.cc:1261] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260805 22:24:54.908113 2254 raft_consensus.cc:3032] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:55.050400 2796 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:55.104982 31357 catalog_manager.cc:5495] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:55.005548 2729 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:55.052687 2616 raft_consensus.cc:686] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 7 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:54.994478 2643 consensus_queue.cc:260] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.039474 2545 consensus_queue.cc:1046] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.114s
I20260805 22:24:55.055049 2383 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260805 22:24:55.150725 31704 raft_consensus.cc:1261] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
W20260805 22:24:55.151183 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 07310455e35e48f8ba4eb9b72a69113b, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:55.149957 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 133611us
I20260805 22:24:55.105612 2731 raft_consensus.cc:2781] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:55.204375 32591 raft_consensus.cc:2454] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:55.039096 2582 consensus_queue.cc:237] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:55.205324 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 07310455e35e48f8ba4eb9b72a69113b, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.057606 2465 raft_consensus.cc:686] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:55.055320 2787 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:55.205806 31667 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:55.223621 31447 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:55.093899 2746 consensus_queue.cc:1046] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.128s
W20260805 22:24:55.098136 2745 consensus_meta.cc:287] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.501s user 0.013s sys 0.000s
I20260805 22:24:55.055384 2773 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:55.063097 2656 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:55.225274 31357 catalog_manager.cc:5495] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 1 to 4. New cstate: current_term: 4 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:24:55.245944 31357 catalog_manager.cc:5495] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 2 to 4, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 4 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:24:55.246999 2445 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:55.222442 2648 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:55.247795 2612 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.248976 31499 raft_consensus.cc:2454] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:55.156314 2636 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:55.249776 2636 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.251955 2636 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:55.103444 2706 raft_consensus.cc:3037] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:55.256139 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
W20260805 22:24:55.257824 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 200870us
I20260805 22:24:55.259033 31554 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:24:55.259351 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 10de517cc0a24f20b5885091e99e252d, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:55.259660 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 10de517cc0a24f20b5885091e99e252d, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.233008 31667 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:55.106279 2609 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:55.226617 2796 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.093497 1989 consensus_queue.cc:1046] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.171s
I20260805 22:24:55.104017 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:55.228012 2731 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:55.237948 2616 consensus_queue.cc:237] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.239470 2748 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:55.149637 2757 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:55.167232 2617 raft_consensus.cc:2781] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:55.242246 2276 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:55.254133 2712 raft_consensus.cc:3032] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Stepping down as leader of term 7
I20260805 22:24:55.103497 2751 raft_consensus.cc:3037] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:55.186236 2637 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:55.202071 2786 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:55.201872 2254 raft_consensus.cc:729] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:55.227921 2607 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:55.315973 2607 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:55.316227 2254 consensus_queue.cc:260] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.243754 2729 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:55.302021 2797 raft_consensus.cc:2781] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:55.317842 2797 raft_consensus.cc:686] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:55.318032 2607 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:55.318504 2612 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:55.318727 2797 consensus_queue.cc:237] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.281297 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
W20260805 22:24:55.294175 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 180285us
I20260805 22:24:55.239615 2750 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:55.328150 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:55.332134 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:55.326009 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:55.276381 2749 raft_consensus.cc:2781] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:55.234380 2782 raft_consensus.cc:2726] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260805 22:24:55.308435 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 8abfbdc4afff482ebc81c4fda5d0812a, TS 516cb07ce3064ccbbca35e912d62816c -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.308269 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:55.307199 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:55.311652 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:55.301331 2617 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:55.312536 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:55.303478 2712 raft_consensus.cc:729] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:55.313872 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:55.313690 2786 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.314491 31490 raft_consensus.cc:1261] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:24:55.314363 2465 consensus_queue.cc:237] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.322604 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:55.277022 2731 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:55.313455 2796 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:55.333454 31715 raft_consensus.cc:1261] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260805 22:24:55.351935 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:55.431245 31603 raft_consensus.cc:2379] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 7: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:55.297478 31502 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:55.332773 31602 raft_consensus.cc:1261] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:24:55.335769 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:55.302848 2609 raft_consensus.cc:2781] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:55.351123 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:55.301525 2445 raft_consensus.cc:3032] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:24:55.302286 2718 raft_consensus.cc:2781] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:55.302222 2752 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:55.313323 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:55.429497 2802 log.cc:924] Time spent T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.095s user 0.001s sys 0.000s
I20260805 22:24:55.403110 31554 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:55.375213 31556 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:55.375768 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:55.437989 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:55.449241 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:55.438382 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 8abfbdc4afff482ebc81c4fda5d0812a, TS 516cb07ce3064ccbbca35e912d62816c -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.437704 31493 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:55.440162 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:55.450893 2656 consensus_queue.cc:1046] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260805 22:24:55.451331 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:55.451468 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:55.452064 2812 raft_consensus.cc:3032] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:24:55.452463 2812 raft_consensus.cc:729] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:55.452648 31713 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:24:55.412933 32585 raft_consensus.cc:2396] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 6 because replica has last-logged OpId of term: 3 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260805 22:24:55.350589 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:55.439714 31600 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:55.445524 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:55.446738 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:24:55.461452 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 127138us
I20260805 22:24:55.463683 31704 raft_consensus.cc:1261] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260805 22:24:55.334837 2749 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:55.452728 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:55.453259 2812 consensus_queue.cc:260] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:55.478796 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:55.440268 31492 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:55.454911 1989 consensus_queue.cc:1046] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 6.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260805 22:24:55.455914 2746 consensus_queue.cc:1046] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:24:55.465736 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:55.469432 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 94332us
I20260805 22:24:55.446385 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:55.485956 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:55.481971 31667 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:55.437443 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:55.474994 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:55.480229 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet f8863c96c162438983f403d38b11c5db, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:55.488389 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet f8863c96c162438983f403d38b11c5db, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.476395 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:55.486055 31601 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:24:55.444254 2617 raft_consensus.cc:3037] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:55.324111 2604 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.491590 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:55.492159 2812 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:55.492743 2800 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:24:55.473368 2718 raft_consensus.cc:686] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:55.495110 2798 consensus_queue.cc:1035] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:24:55.497735 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:55.438961 2445 raft_consensus.cc:729] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:55.499400 31713 raft_consensus.cc:1261] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260805 22:24:55.505705 31608 raft_consensus.cc:1226] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:55.483892 2749 raft_consensus.cc:3037] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:55.525699 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:55.508035 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:55.509516 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:55.375061 2383 consensus_queue.cc:1035] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.105s
I20260805 22:24:55.350914 2174 consensus_queue.cc:1046] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.127s
I20260805 22:24:55.448308 2716 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:55.522614 31667 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
W20260805 22:24:55.523192 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 357679us
I20260805 22:24:55.482735 2757 raft_consensus.cc:2726] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:24:55.527395 31707 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:55.532637 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:24:55.555787 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 44703us
I20260805 22:24:55.521447 2800 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:55.557485 31667 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:55.559340 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:55.559695 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:55.566263 31554 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:55.566316 31667 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:55.568338 31554 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:55.524667 2520 raft_consensus.cc:2726] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:24:55.574689 31667 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:55.578490 31667 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:55.580261 31596 raft_consensus.cc:1226] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:55.562161 2782 raft_consensus.cc:2726] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:55.596352 31444 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:55.597448 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:55.558451 2800 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:24:55.569850 2609 raft_consensus.cc:686] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:55.447795 2752 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:55.578007 2766 raft_consensus.cc:2781] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:55.555451 2795 consensus_queue.cc:1035] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.084s
I20260805 22:24:55.578233 2716 raft_consensus.cc:2781] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election won for term 7
W20260805 22:24:55.667593 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 65997.9us
I20260805 22:24:55.604751 32579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57382 (request call id 665) took 1220 ms. Trace:
I20260805 22:24:55.582477 2637 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:24:55.688532 2637 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:55.689060 2637 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:55.506091 2712 consensus_queue.cc:260] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.589802 2739 raft_consensus.cc:2726] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:24:55.592163 2646 raft_consensus.cc:2781] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:55.595736 2745 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:55.596946 2628 raft_consensus.cc:2781] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:55.708949 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:55.677232 2766 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:55.606143 2797 consensus_queue.cc:1035] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.104s
I20260805 22:24:55.570114 2729 consensus_queue.cc:1035] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.107s
I20260805 22:24:55.695393 2718 consensus_queue.cc:237] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 3.3, Last appended by leader: 3, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:55.677112 2716 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:55.710248 2766 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:55.746982 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:55.760528 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 80745.7us
W20260805 22:24:55.762859 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 171754us
I20260805 22:24:55.707574 2646 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:55.767513 31667 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:55.798825 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 130391us
I20260805 22:24:55.716636 2628 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:55.599550 2811 consensus_queue.cc:1046] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: INVALID_TERM, Last received: 6.7, Next index: 8, Last known committed idx: 6, Time since last communication: 0.163s
I20260805 22:24:55.809768 31554 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:55.768340 32579 rpcz_store.cc:276] 0805 22:24:54.379291 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:54.393500 (+ 14209us) service_pool.cc:227] Handling call
0805 22:24:54.507511 (+114011us) spinlock_profiling.cc:243] Waited 8.41 ms on lock 0x7b5800005198. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735faa1b0 00007f2735fa9a94 00007f272f691c04 00007f272f691b92
0805 22:24:55.592501 (+1084990us) spinlock_profiling.cc:243] Waited 89.2 ms on lock 0x7b540007eed4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:55.600095 (+ 7594us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":224395264}
I20260805 22:24:55.745528 2716 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:55.695694 2748 consensus_queue.cc:1035] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.062s
I20260805 22:24:55.821978 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:55.815299 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:55.891075 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:55.894393 31495 raft_consensus.cc:2379] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 7: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:55.888252 31447 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:55.894459 31554 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:55.729784 2745 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:55.901299 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 133873us
I20260805 22:24:55.806905 2520 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:55.801323 2646 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:55.908241 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:55.910895 31501 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:55.812242 32583 raft_consensus.cc:1261] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260805 22:24:55.816987 2797 raft_consensus.cc:2781] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260805 22:24:55.916663 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 320393us
I20260805 22:24:55.810923 2607 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:55.812361 2276 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:24:55.811242 2628 raft_consensus.cc:3037] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
W20260805 22:24:55.986546 2780 log.cc:924] Time spent T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20260805 22:24:55.911707 2782 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:24:56.014845 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 114099us
I20260805 22:24:55.915388 2616 raft_consensus.cc:2726] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:24:56.032114 31501 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:55.811939 2609 consensus_queue.cc:237] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:55.920428 2604 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:55.922386 2797 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:56.015074 2786 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:56.070992 2786 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:56.071532 2786 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.074061 2824 consensus_queue.cc:1035] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.335s
I20260805 22:24:56.075707 2228 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:56.076045 2824 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:56.076792 2228 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:56.076570 2766 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.077071 2820 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:56.004637 2520 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.006711 2607 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:56.099463 31501 raft_consensus.cc:1261] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260805 22:24:56.108253 2841 consensus_queue.cc:1035] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.297s
I20260805 22:24:56.109510 2837 raft_consensus.cc:2726] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:24:56.112236 31490 raft_consensus.cc:1226] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5. Ops: [4.3-4.3]
I20260805 22:24:56.150439 31501 raft_consensus.cc:1261] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:24:56.058239 2797 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:56.198323 31501 raft_consensus.cc:1226] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 6. Current term is 7. Ops: []
I20260805 22:24:56.080026 2750 raft_consensus.cc:2781] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:56.208927 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:56.036262 2782 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:56.240299 31444 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:56.240789 31554 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:56.249790 31444 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:56.251302 31444 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:56.077629 2820 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.256263 2820 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:56.258463 2820 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:56.260026 2820 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.263947 2820 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:56.270109 2820 raft_consensus.cc:2726] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:56.271725 2820 raft_consensus.cc:2726] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260805 22:24:56.272436 2820 consensus_queue.cc:1046] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:24:56.113854 2637 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:56.120010 2228 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.062920 2828 consensus_queue.cc:1046] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 2, Time since last communication: 0.585s
I20260805 22:24:56.141196 2842 consensus_queue.cc:1035] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.106s
I20260805 22:24:56.218007 2750 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:56.233539 2773 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:56.299355 2855 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
W20260805 22:24:56.291711 2437 consensus_meta.cc:287] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 1.354s user 0.014s sys 0.001s
I20260805 22:24:56.307817 2437 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.311148 31492 raft_consensus.cc:1226] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:56.076618 2824 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.300030 2855 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.310472 31499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40740 (request call id 1231) took 1775 ms. Trace:
I20260805 22:24:56.288594 2750 raft_consensus.cc:3037] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
W20260805 22:24:56.315114 2856 log.cc:924] Time spent T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.103s user 0.000s sys 0.001s
W20260805 22:24:56.315975 2731 consensus_meta.cc:287] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.875s user 0.005s sys 0.006s
I20260805 22:24:56.310607 2437 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:56.327587 2520 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:56.329285 2607 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:56.317185 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:56.291072 2820 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:56.334609 2782 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:56.336236 2766 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:56.338397 2637 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:56.327200 2228 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:56.343604 31499 rpcz_store.cc:276] 0805 22:24:54.534984 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:54.535395 (+ 411us) service_pool.cc:227] Handling call
0805 22:24:56.310211 (+1774816us) spinlock_profiling.cc:243] Waited 533 ms on lock 0x7b5400121954. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:56.310341 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1226097920}
I20260805 22:24:56.340991 2636 raft_consensus.cc:3032] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:24:56.296248 2786 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:56.398859 2636 raft_consensus.cc:729] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
W20260805 22:24:56.399768 2804 log.cc:924] Time spent T 4172338662524cb29310f48e0861873e P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 1.112s user 0.004s sys 0.000s
I20260805 22:24:56.402825 2728 raft_consensus.cc:3032] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:24:56.403322 2728 raft_consensus.cc:729] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:56.298569 2745 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:56.359270 2814 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:56.360466 31708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 2230) took 1752 ms. Trace:
I20260805 22:24:56.446347 31708 rpcz_store.cc:276] 0805 22:24:54.607625 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:54.715019 (+107394us) service_pool.cc:227] Handling call
0805 22:24:54.715272 (+ 253us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:54.895649 (+180377us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:54.895680 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:54.895719 (+ 39us) raft_consensus.cc:1631] Marking committed up to 1
0805 22:24:54.895748 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:56.360209 (+1464461us) spinlock_profiling.cc:243] Waited 180 ms on lock 0x7b5400218c54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:56.360259 (+ 50us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:56.360404 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":414405888}
I20260805 22:24:56.337024 2445 consensus_queue.cc:260] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.363765 2616 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:56.448065 2616 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.452526 2602 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:56.366873 32589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57372 (request call id 640) took 1310 ms. Trace:
I20260805 22:24:56.367331 2824 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:56.353436 2731 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:56.370772 2767 raft_consensus.cc:3032] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:24:56.459084 2767 raft_consensus.cc:729] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:56.459326 2616 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:56.372673 2749 consensus_meta.cc:287] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.810s user 0.008s sys 0.008s
W20260805 22:24:56.376147 2706 consensus_meta.cc:287] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 1.107s user 0.019s sys 0.002s
I20260805 22:24:56.375835 2383 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:56.295106 2773 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.380172 2705 raft_consensus.cc:2726] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:24:56.388406 31706 raft_consensus.cc:1226] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 6. Current term is 7. Ops: []
I20260805 22:24:56.392747 31711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 2242) took 1738 ms. Trace:
W20260805 22:24:56.387890 31709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1798) took 1795 ms (client timeout 2220 ms). Trace:
I20260805 22:24:56.398483 31499 raft_consensus.cc:1261] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
W20260805 22:24:56.408473 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 63317.6us
I20260805 22:24:56.400049 2636 consensus_queue.cc:260] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:56.404435 2728 consensus_queue.cc:260] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:56.413164 2855 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:56.389987 2437 consensus_queue.cc:1035] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.025s
I20260805 22:24:56.415838 2646 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:56.427114 2254 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:56.460587 2767 consensus_queue.cc:260] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:56.431843 2751 consensus_meta.cc:287] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 1.122s user 0.002s sys 0.030s
W20260805 22:24:56.439217 2617 consensus_meta.cc:287] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.913s user 0.013s sys 0.013s
I20260805 22:24:56.457530 32589 rpcz_store.cc:276] 0805 22:24:55.056452 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:55.056662 (+ 210us) service_pool.cc:227] Handling call
0805 22:24:56.366154 (+1309492us) spinlock_profiling.cc:243] Waited 2 ms on lock 0x7b5800005198. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:24:56.366367 (+ 213us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:56.366565 (+ 198us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:56.366592 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:56.366631 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:24:56.366664 (+ 33us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:56.366719 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:56.366818 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4607872}
W20260805 22:24:56.458304 2776 log.cc:924] Time spent T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 1.461s user 0.005s sys 0.000s
I20260805 22:24:56.692759 31608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56384 (request call id 370) took 1129 ms. Trace:
I20260805 22:24:56.687028 31714 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 2232) took 2078 ms. Trace:
I20260805 22:24:56.692888 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:56.691833 31604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59030 (request call id 1216) took 1384 ms. Trace:
I20260805 22:24:56.665231 31602 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1231) took 1090 ms. Trace:
I20260805 22:24:56.707696 31604 rpcz_store.cc:276] 0805 22:24:55.307208 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:55.307344 (+ 136us) service_pool.cc:227] Handling call
0805 22:24:56.691488 (+1384144us) spinlock_profiling.cc:243] Waited 480 ms on lock 0x7b54002387d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:56.691685 (+ 197us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1103654272}
I20260805 22:24:56.522588 31500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40740 (request call id 1233) took 1214 ms. Trace:
W20260805 22:24:56.621937 31447 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.010s sys 0.078s
W20260805 22:24:56.709197 31447 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.673s user 0.010s sys 0.088s
I20260805 22:24:56.708078 31602 rpcz_store.cc:276] 0805 22:24:55.575002 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:55.575641 (+ 639us) service_pool.cc:227] Handling call
0805 22:24:55.599860 (+ 24219us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:55.717253 (+117393us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:55.717285 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:55.717324 (+ 39us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:24:55.717353 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:56.664950 (+947597us) spinlock_profiling.cc:243] Waited 117 ms on lock 0x7b54001437d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:56.665003 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:56.665162 (+ 159us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":269475072}
I20260805 22:24:56.708807 31500 rpcz_store.cc:276] 0805 22:24:55.308162 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:55.308713 (+ 551us) service_pool.cc:227] Handling call
0805 22:24:55.308963 (+ 250us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:55.477153 (+168190us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:55.477182 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:55.477219 (+ 37us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:24:55.477248 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:56.522344 (+1045096us) spinlock_profiling.cc:243] Waited 168 ms on lock 0x7b5400121e54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:56.522390 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:56.522522 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":386381696}
I20260805 22:24:56.702530 31608 rpcz_store.cc:276] 0805 22:24:55.563236 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:55.563393 (+ 157us) service_pool.cc:227] Handling call
0805 22:24:55.563727 (+ 334us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:55.564066 (+ 339us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:55.564094 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:55.592903 (+ 28809us) log.cc:842] Serialized 26 byte log entry
0805 22:24:55.805880 (+212977us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:24:55.805917 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:55.805985 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:56.692526 (+886541us) raft_consensus.cc:1671] finished
0805 22:24:56.692546 (+ 20us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:56.692696 (+ 150us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":13891,"raft.queue_time_us":604,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":7973,"thread_start_us":130899,"threads_started":1,"wal-append.queue_time_us":1057334}
I20260805 22:24:56.713485 31705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 2228) took 2124 ms. Trace:
I20260805 22:24:56.713835 31705 rpcz_store.cc:276] 0805 22:24:54.589354 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:54.589631 (+ 277us) service_pool.cc:227] Handling call
0805 22:24:54.589973 (+ 342us) raft_consensus.cc:1457] Updating replica for 2 ops
0805 22:24:54.605809 (+ 15836us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:54.605840 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0805 22:24:54.606461 (+ 621us) log.cc:842] Serialized 52 byte log entry
0805 22:24:54.857906 (+251445us) raft_consensus.cc:1631] Marking committed up to 7
0805 22:24:54.877076 (+ 19170us) log.cc:842] Serialized 14 byte log entry
0805 22:24:54.877517 (+ 441us) log.cc:842] Serialized 14 byte log entry
0805 22:24:54.877720 (+ 203us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:54.877788 (+ 68us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:56.360377 (+1482589us) spinlock_profiling.cc:243] Waited 6.74 ms on lock 0x7b180041c560. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e140638 00007f272e143b11 00007f272e143ac9 00007f272e143a59 00007f272e143a21 00007f272e142d1d 00007f27373a4204 00007f27373a4039 00007f272e122dc6
0805 22:24:56.455544 (+ 95167us) spinlock_profiling.cc:243] Waited 3.25 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:24:56.713221 (+257677us) raft_consensus.cc:1671] finished
0805 22:24:56.713245 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:56.713393 (+ 148us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":28879,"prepare.queue_time_us":49302,"prepare.run_cpu_time_us":80,"prepare.run_wall_time_us":82,"raft.queue_time_us":23003,"raft.run_cpu_time_us":459,"raft.run_wall_time_us":55821,"spinlock_wait_cycles":22986240,"thread_start_us":251090,"threads_started":1,"wal-append.queue_time_us":377853}
I20260805 22:24:56.529222 1989 raft_consensus.cc:3032] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Stepping down as leader of term 2
W20260805 22:24:56.709631 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 674034us
I20260805 22:24:56.716486 1989 raft_consensus.cc:729] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:56.690920 2712 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:56.719496 31594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59030 (request call id 1212) took 1413 ms. Trace:
I20260805 22:24:56.726307 31494 raft_consensus.cc:2454] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:56.726656 31594 rpcz_store.cc:276] 0805 22:24:55.306067 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:55.306238 (+ 171us) service_pool.cc:227] Handling call
0805 22:24:56.692714 (+1386476us) spinlock_profiling.cc:243] Waited 3.31 ms on lock 0x7b5800004598. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735faa1b0 00007f2735fa9a94 00007f272f691c04 00007f272f691b92
0805 22:24:56.719431 (+ 26717us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":7603584}
I20260805 22:24:56.702809 31714 rpcz_store.cc:276] 0805 22:24:54.608127 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:54.716350 (+108223us) service_pool.cc:227] Handling call
0805 22:24:55.249668 (+533318us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:55.249871 (+ 203us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:55.249893 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:55.249934 (+ 41us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:24:55.249964 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:55.250016 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:56.686822 (+1436806us) spinlock_profiling.cc:243] Waited 533 ms on lock 0x7b5400292cd0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731067a09 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:56.686962 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1225685248}
I20260805 22:24:56.743095 32586 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
W20260805 22:24:56.702036 31709 rpcz_store.cc:269] 0805 22:24:54.592916 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:54.593483 (+ 567us) service_pool.cc:227] Handling call
0805 22:24:56.387701 (+1794218us) spinlock_profiling.cc:243] Waited 75.2 ms on lock 0x7b54001e2b54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:56.387830 (+ 129us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":172915072}
W20260805 22:24:56.686792 2846 log.cc:924] Time spent T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.601s user 0.002s sys 0.000s
W20260805 22:24:56.644757 2812 consensus_meta.cc:287] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 1.103s user 0.010s sys 0.009s
W20260805 22:24:56.512491 2794 log.cc:924] Time spent T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 1.246s user 0.001s sys 0.001s
I20260805 22:24:56.720283 2706 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:56.746358 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:56.607560 2747 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:56.747840 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:56.748523 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:56.784235 32587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57366 (request call id 456) took 2095 ms. Trace:
I20260805 22:24:56.749018 31492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1776) took 1179 ms. Trace:
I20260805 22:24:56.786123 31492 rpcz_store.cc:276] 0805 22:24:55.569319 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:55.914612 (+345293us) service_pool.cc:227] Handling call
0805 22:24:55.914878 (+ 266us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:56.330651 (+415773us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:56.748771 (+418120us) spinlock_profiling.cc:243] Waited 396 ms on lock 0x7b54002a0b54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:56.748954 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":910911872}
I20260805 22:24:56.462393 2824 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:24:56.532119 2716 consensus_meta.cc:287] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.614s user 0.009s sys 0.003s
I20260805 22:24:56.570420 2665 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:56.678859 2670 raft_consensus.cc:3032] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:56.688701 2643 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:56.754750 31715 raft_consensus.cc:1226] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 5. Current term is 6. Ops: []
W20260805 22:24:56.661649 2840 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.031s sys 0.057s
I20260805 22:24:56.723511 2631 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:56.748924 32584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57382 (request call id 672) took 1808 ms. Trace:
I20260805 22:24:56.718384 2797 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:56.758167 31448 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.635s user 0.020s sys 0.034s
I20260805 22:24:56.499955 2666 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:56.821717 2631 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.747184 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:56.465991 2276 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:56.753892 2656 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:56.825780 2863 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:24:56.826808 31448 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.817s user 0.021s sys 0.035s
I20260805 22:24:56.826402 2656 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:56.827831 2840 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.775s user 0.034s sys 0.062s
I20260805 22:24:56.822418 32584 rpcz_store.cc:276] 0805 22:24:54.940503 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:54.951923 (+ 11420us) service_pool.cc:227] Handling call
0805 22:24:56.747816 (+1795893us) spinlock_profiling.cc:243] Waited 13.4 ms on lock 0x7b5800005198. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:24:56.748158 (+ 342us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:56.748423 (+ 265us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:56.748458 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:56.748507 (+ 49us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:24:56.748598 (+ 91us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:56.748700 (+ 102us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:56.748840 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":30766464}
I20260805 22:24:56.751714 2751 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:56.771286 2646 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:56.771744 2636 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:56.829694 2751 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:56.637226 2616 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:56.461254 2731 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:56.730995 2728 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:56.719460 2749 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.743498 31711 rpcz_store.cc:276] 0805 22:24:54.654742 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:55.053529 (+398787us) service_pool.cc:227] Handling call
0805 22:24:55.097137 (+ 43608us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:55.152597 (+ 55460us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:55.152627 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:55.152678 (+ 51us) raft_consensus.cc:1631] Marking committed up to 2
0805 22:24:55.152708 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:56.392494 (+1239786us) spinlock_profiling.cc:243] Waited 55.2 ms on lock 0x7b5400150754. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:56.392541 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:56.392687 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":127062272}
I20260805 22:24:56.515565 2752 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:56.763545 2773 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:24:56.528190 31502 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1757) took 1967 ms (client timeout 2086 ms). Trace:
I20260805 22:24:56.784660 32587 rpcz_store.cc:276] 0805 22:24:54.688602 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:54.688879 (+ 277us) service_pool.cc:227] Handling call
0805 22:24:54.826337 (+137458us) spinlock_profiling.cc:243] Waited 2.11 ms on lock 0x7b5800005198. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:24:54.826553 (+ 216us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:54.870212 (+ 43659us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:54.870247 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:54.872701 (+ 2454us) log.cc:842] Serialized 26 byte log entry
0805 22:24:55.001970 (+129269us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:24:55.002010 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:55.002076 (+ 66us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:56.473879 (+1471803us) spinlock_profiling.cc:243] Waited 16.8 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:24:56.784019 (+310140us) raft_consensus.cc:1671] finished
0805 22:24:56.784041 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:56.784178 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":43218,"raft.queue_time_us":45009,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":20706,"spinlock_wait_cycles":43585536,"thread_start_us":128927,"threads_started":1,"wal-append.queue_time_us":224854}
I20260805 22:24:56.675334 2531 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
W20260805 22:24:56.532505 2800 consensus_meta.cc:287] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.848s user 0.015s sys 0.016s
W20260805 22:24:56.793921 2816 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.740s user 0.007s sys 0.037s
I20260805 22:24:56.845011 2616 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:56.860353 2816 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.874s user 0.007s sys 0.038s
I20260805 22:24:56.797592 31494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40740 (request call id 1235) took 1488 ms. Trace:
W20260805 22:24:56.798403 32586 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57372 (request call id 646) took 1700 ms (client timeout 1509 ms). Trace:
I20260805 22:24:56.468786 2628 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:56.787060 31597 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.878s user 0.027s sys 0.083s
I20260805 22:24:56.826712 2631 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:56.825098 2666 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.833809 2797 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:56.719818 2383 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.825903 2276 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:24:56.809291 2670 raft_consensus.cc:729] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:56.819193 2796 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:56.861270 2800 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:56.734164 2617 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:56.857642 2706 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:56.842927 31706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 2236) took 2228 ms. Trace:
I20260805 22:24:56.924791 31706 rpcz_store.cc:276] 0805 22:24:54.614673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:54.725925 (+111252us) service_pool.cc:227] Handling call
0805 22:24:56.388208 (+1662283us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:56.709321 (+321113us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:56.842700 (+133379us) spinlock_profiling.cc:243] Waited 1.66 s on lock 0x7b54001e2b50. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731067a09 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:56.842867 (+ 167us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3822185216}
I20260805 22:24:56.854331 31493 raft_consensus.cc:2379] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 4: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:56.855505 2731 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:56.929229 2763 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:56.928755 2731 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:56.857568 31502 rpcz_store.cc:269] 0805 22:24:54.560669 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:54.564353 (+ 3684us) service_pool.cc:227] Handling call
0805 22:24:56.528015 (+1963662us) spinlock_profiling.cc:243] Waited 688 ms on lock 0x7b54002c15d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:56.528128 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1583429376}
I20260805 22:24:56.837261 2646 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:56.858958 2531 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:56.836819 2787 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:56.862301 31494 rpcz_store.cc:276] 0805 22:24:55.308709 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:55.308832 (+ 123us) service_pool.cc:227] Handling call
0805 22:24:56.797423 (+1488591us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b54001fd454. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:56.797533 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3204748160}
W20260805 22:24:56.863039 32586 rpcz_store.cc:269] 0805 22:24:55.098359 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:55.151254 (+ 52895us) service_pool.cc:227] Handling call
0805 22:24:56.316984 (+1165730us) spinlock_profiling.cc:243] Waited 17.4 ms on lock 0x7b5800005198. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735faa1b0 00007f2735fa9a94 00007f272f691c04 00007f272f691b92
0805 22:24:56.798230 (+481246us) spinlock_profiling.cc:243] Waited 388 ms on lock 0x7b54004d0b54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:56.798335 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":931909120}
I20260805 22:24:56.826505 2863 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:56.767058 2808 log.cc:924] Time spent T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 1.209s user 0.000s sys 0.002s
I20260805 22:24:56.875752 31492 raft_consensus.cc:1261] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:56.891211 2628 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:56.892177 2656 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:56.894832 32581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57382 (request call id 679) took 1309 ms. Trace:
W20260805 22:24:56.897953 2820 consensus_meta.cc:287] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.503s user 0.018s sys 0.005s
I20260805 22:24:56.900249 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:24:56.790704 2747 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.926548 2616 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:56.798792 2824 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.810389 2716 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:56.462656 2814 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.787281 2812 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:56.894604 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
W20260805 22:24:56.879676 31597 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.970s user 0.027s sys 0.083s
I20260805 22:24:56.785305 2437 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:56.858989 2749 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:56.832864 1989 consensus_queue.cc:260] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.961653 2820 raft_consensus.cc:2726] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:24:56.821264 2665 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.942308 2842 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:56.971684 31493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:52098 (request call id 410) took 1523 ms. Trace:
I20260805 22:24:57.009604 31493 rpcz_store.cc:276] 0805 22:24:55.448475 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:55.491275 (+ 42800us) service_pool.cc:227] Handling call
0805 22:24:56.971246 (+1479971us) spinlock_profiling.cc:243] Waited 1.35 s on lock 0x7b54002e5754. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:56.971390 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3098964352}
I20260805 22:24:56.923650 2796 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:56.981784 31713 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.681s user 0.011s sys 0.018s
W20260805 22:24:56.910183 31668 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.321s user 0.035s sys 0.095s
I20260805 22:24:57.007339 2666 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:56.929949 2763 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.975049 2767 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Advancing to term 7
W20260805 22:24:57.017524 31713 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.716s user 0.012s sys 0.018s
W20260805 22:24:57.017601 31668 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.428s user 0.035s sys 0.095s
I20260805 22:24:56.950433 2646 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.978636 2751 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:57.003371 2609 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.862s user 0.011s sys 0.044s
W20260805 22:24:57.002066 2465 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.277s user 0.008s sys 0.042s
I20260805 22:24:56.993240 32585 raft_consensus.cc:3037] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:24:57.015082 2750 consensus_meta.cc:287] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.660s user 0.016s sys 0.004s
I20260805 22:24:56.949892 2787 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:57.014349 2762 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.631s user 0.003s sys 0.043s
I20260805 22:24:57.039984 2452 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:57.041754 2855 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:57.042301 2763 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:56.994879 2800 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:57.043422 2465 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.328s user 0.008s sys 0.042s
W20260805 22:24:57.037312 2609 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.896s user 0.011s sys 0.044s
I20260805 22:24:56.933334 2670 consensus_queue.cc:260] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:57.065778 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:57.029888 2797 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:56.981182 2812 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:56.994535 2437 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:57.044433 1989 raft_consensus.cc:3037] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:57.033565 2820 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
W20260805 22:24:57.072831 32529 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.515s user 0.015s sys 0.035s
W20260805 22:24:57.124069 32529 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.569s user 0.015s sys 0.035s
I20260805 22:24:57.044281 31597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1211) took 1738 ms. Trace:
I20260805 22:24:57.079133 32594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57366 (request call id 462) took 1809 ms. Trace:
W20260805 22:24:57.076903 31603 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.831s user 0.025s sys 0.027s
W20260805 22:24:57.125139 31603 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.886s user 0.025s sys 0.035s
W20260805 22:24:57.059959 2762 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.677s user 0.004s sys 0.043s
I20260805 22:24:57.027583 2842 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:57.047793 31667 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.143s user 0.002s sys 0.017s
I20260805 22:24:57.121696 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:56.959430 32581 rpcz_store.cc:276] 0805 22:24:55.585056 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:55.590460 (+ 5404us) service_pool.cc:227] Handling call
0805 22:24:56.758377 (+1167917us) spinlock_profiling.cc:243] Waited 9.36 ms on lock 0x7b5800005198. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108bce 000000000034aea9 00007f2735f34a30 00007f2735ffa7b4 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:24:56.819030 (+ 60653us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:56.821709 (+ 2679us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:56.821759 (+ 50us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:56.821807 (+ 48us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:24:56.821845 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:56.821909 (+ 64us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:56.894462 (+ 72553us) spinlock_profiling.cc:243] Waited 49.8 ms on lock 0x7b54004d0b50. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731067a09 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:56.894727 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":136007936}
I20260805 22:24:57.124902 32594 rpcz_store.cc:276] 0805 22:24:55.269238 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:56.385622 (+1116384us) service_pool.cc:227] Handling call
0805 22:24:56.385988 (+ 366us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:57.004944 (+618956us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:57.004975 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:57.005013 (+ 38us) raft_consensus.cc:1631] Marking committed up to 7
0805 22:24:57.005043 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:57.078895 (+ 73852us) spinlock_profiling.cc:243] Waited 619 ms on lock 0x7b540017cad4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:24:57.078942 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:57.079067 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1422982400}
I20260805 22:24:57.104517 2800 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:57.060735 2750 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:57.072418 2855 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:57.076607 2452 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:57.124435 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 1.64049e+06us
W20260805 22:24:57.181672 31501 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.940s user 0.002s sys 0.020s
I20260805 22:24:57.127730 31603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1223) took 1692 ms. Trace:
I20260805 22:24:57.131699 32591 raft_consensus.cc:3037] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
W20260805 22:24:57.160493 31554 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.907s user 0.001s sys 0.019s
W20260805 22:24:57.515733 31667 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.613s user 0.017s sys 0.022s
I20260805 22:24:57.519471 31667 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:57.516652 31501 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.275s user 0.008s sys 0.031s
W20260805 22:24:57.518224 31554 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.266s user 0.002s sys 0.034s
W20260805 22:24:57.521238 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 1.27311e+06us
W20260805 22:24:57.180272 31450 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.170s user 0.023s sys 0.014s
W20260805 22:24:57.155037 31561 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.934s user 0.015s sys 0.033s
I20260805 22:24:57.525027 31554 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
W20260805 22:24:57.525485 31450 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.517s user 0.035s sys 0.029s
I20260805 22:24:57.117750 2656 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:57.528746 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:57.120844 2465 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:24:57.570063 31561 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.534s user 0.016s sys 0.051s
W20260805 22:24:57.180779 2871 log.cc:924] Time spent T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0: Append to log took a long time: real 0.089s user 0.001s sys 0.001s
W20260805 22:24:57.567591 31556 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.328s user 0.032s sys 0.045s
I20260805 22:24:57.537453 31667 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:57.234406 2545 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 2.003s user 0.006s sys 0.027s
W20260805 22:24:57.575027 2545 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 2.346s user 0.007s sys 0.036s
I20260805 22:24:57.147053 2531 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:57.120126 2617 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:57.570278 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:57.521005 32584 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.628s user 0.014s sys 0.011s
I20260805 22:24:57.142513 2670 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:24:57.351593 2882 log.cc:924] Time spent T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.148s user 0.000s sys 0.001s
I20260805 22:24:57.517477 31603 rpcz_store.cc:276] 0805 22:24:55.435014 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:55.435120 (+ 106us) service_pool.cc:227] Handling call
0805 22:24:55.442655 (+ 7535us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:55.442850 (+ 195us) raft_consensus.cc:1520] Early marking committed up to index 7
0805 22:24:55.892010 (+449160us) log.cc:842] Serialized 14 byte log entry
0805 22:24:57.127377 (+1235367us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:57.127435 (+ 58us) raft_consensus.cc:1631] Marking committed up to 7
0805 22:24:57.127482 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:57.127551 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:57.127673 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24495,"prepare.queue_time_us":85745,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":164863,"raft.run_cpu_time_us":89,"raft.run_wall_time_us":99,"thread_start_us":1659240,"threads_started":3,"wal-append.queue_time_us":766948}
I20260805 22:24:57.105131 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:57.574206 31556 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.335s user 0.033s sys 0.045s
I20260805 22:24:57.516041 2445 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Advancing to term 7
W20260805 22:24:57.514714 2254 consensus_meta.cc:287] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.772s user 0.006s sys 0.006s
I20260805 22:24:57.194069 2747 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:57.459679 2276 consensus_meta.cc:287] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.552s user 0.010s sys 0.001s
I20260805 22:24:57.211114 2812 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:57.205493 2863 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:57.447168 2869 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:57.270012 2665 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:57.255676 2646 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:57.195851 2731 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:57.714748 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:57.528715 2824 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:57.371601 2437 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:57.522486 2609 raft_consensus.cc:2726] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:24:57.344908 2763 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:57.285347 2631 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:57.195649 2716 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:57.141673 2797 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:57.129454 2820 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:57.575307 2857 raft_consensus.cc:2726] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:24:57.503312 2814 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:57.522926 2842 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:57.570497 2867 raft_consensus.cc:2781] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:57.705013 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:57.528100 2800 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:57.572485 2656 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:57.706836 2796 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:57.584318 2465 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:57.713439 2864 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:57.126827 2383 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:57.726003 2628 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:57.688551 2876 consensus_queue.cc:1035] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.061s
I20260805 22:24:57.738102 2863 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:57.749184 2631 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:57.732276 2276 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:57.753013 2712 consensus_meta.cc:287] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 1.035s user 0.006s sys 0.009s
I20260805 22:24:57.767131 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:57.740538 2877 consensus_queue.cc:1035] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.022s
W20260805 22:24:57.760250 2636 consensus_meta.cc:287] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.930s user 0.011s sys 0.002s
W20260805 22:24:57.780501 31713 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.732s user 0.071s sys 0.090s
I20260805 22:24:57.747097 2731 consensus_queue.cc:1035] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.028s
W20260805 22:24:57.773728 2728 consensus_meta.cc:287] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.925s user 0.013s sys 0.012s
I20260805 22:24:57.764993 2891 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:57.750475 2824 consensus_queue.cc:1046] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.029s
I20260805 22:24:57.740015 2869 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:57.781312 2787 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:57.752465 2763 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:57.783052 2750 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:57.791373 2751 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:57.800619 2891 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:57.804422 2751 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:57.785168 2867 raft_consensus.cc:686] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:57.785894 2820 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:57.782905 2864 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:57.783111 2628 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:57.796932 2633 raft_consensus.cc:3032] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:24:57.787819 2863 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:57.800640 2855 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:57.801519 2631 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:57.810781 2633 raft_consensus.cc:729] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
W20260805 22:24:57.740068 31423 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Leader has not yet committed an operation in its own term
W20260805 22:24:57.856879 32589 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.663s user 0.007s sys 0.009s
I20260805 22:24:57.802408 2452 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:57.913558 32589 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.727s user 0.007s sys 0.010s
I20260805 22:24:57.915426 2276 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:57.915711 2728 raft_consensus.cc:3032] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:57.916357 2728 raft_consensus.cc:729] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:57.126413 31597 rpcz_store.cc:276] 0805 22:24:55.305749 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:55.310899 (+ 5150us) service_pool.cc:227] Handling call
0805 22:24:55.409641 (+ 98742us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:55.423856 (+ 14215us) raft_consensus.cc:1520] Early marking committed up to index 3
0805 22:24:55.529274 (+105418us) log.cc:842] Serialized 14 byte log entry
0805 22:24:56.472263 (+942989us) spinlock_profiling.cc:243] Waited 4.62 ms on lock 0x7b18003d8640. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e140638 00007f272e143b11 00007f272e143ac9 00007f272e143a59 00007f272e143a21 00007f272e142d1d 00007f27373a4204 00007f27373a4039 00007f272e122dc6
0805 22:24:56.981590 (+509327us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:56.993182 (+ 11592us) raft_consensus.cc:1631] Marking committed up to 3
0805 22:24:57.008466 (+ 15284us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:57.026769 (+ 18303us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:57.038419 (+ 11650us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":96459,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":35,"raft.queue_time_us":73311,"raft.run_cpu_time_us":111,"raft.run_wall_time_us":112,"spinlock_wait_cycles":10622336,"thread_start_us":1399866,"threads_started":3,"wal-append.queue_time_us":497622,"wal-append.run_cpu_time_us":1200,"wal-append.run_wall_time_us":1011072}
W20260805 22:24:57.858470 2762 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.678s user 0.014s sys 0.018s
W20260805 22:24:57.917798 2762 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.761s user 0.014s sys 0.019s
I20260805 22:24:57.860210 2800 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:57.780452 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:57.806380 32591 raft_consensus.cc:2454] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
W20260805 22:24:57.791415 31713 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.743s user 0.071s sys 0.090s
I20260805 22:24:57.822561 2867 consensus_queue.cc:237] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.3, Last appended by leader: 3, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:57.942396 2728 consensus_queue.cc:260] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:57.943931 2728 raft_consensus.cc:3037] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:57.863901 2763 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:57.864197 2869 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:57.948100 2628 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:57.866773 2751 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:57.796849 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:57.949950 31556 connection.cc:232] RPC call timeout handler was delayed by -1.0534700818825513s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18430000000000002s for a response
I20260805 22:24:57.951092 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:57.952188 31556 leader_election.cc:334] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Timed out: RequestConsensusVote RPC to 127.29.21.196:41801 timed out after 1.509s (SENT)
I20260805 22:24:57.952826 31556 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:57.872231 2656 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:57.953795 2884 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:57.954770 2884 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:57.822325 2891 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:57.957772 2731 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:57.957932 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:57.958949 31594 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:57.958649 2731 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:57.959754 2891 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:57.961031 2731 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:57.872146 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:57.895814 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 209360us
I20260805 22:24:57.962023 2800 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:57.962690 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:57.756276 31501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1780) took 1858 ms. Trace:
I20260805 22:24:57.906278 2465 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:57.962975 2800 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:57.906471 2855 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:57.825453 2863 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:57.956988 2876 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:57.908504 2633 consensus_queue.cc:260] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:57.909473 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 325080us
I20260805 22:24:57.962164 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:57.963145 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:24:57.920970 2643 consensus_meta.cc:287] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 1.098s user 0.014s sys 0.002s
W20260805 22:24:57.923756 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 434168us
W20260805 22:24:57.925319 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 387820us
I20260805 22:24:57.920893 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:57.941339 2617 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:57.983415 2617 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:57.983947 2643 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:57.941643 31713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1819) took 1737 ms. Trace:
I20260805 22:24:57.981825 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:57.921329 2797 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:57.974020 2762 raft_consensus.cc:3032] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:57.971813 31604 raft_consensus.cc:2379] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:57.986245 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:57.960574 2891 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:57.963723 31501 rpcz_store.cc:276] 0805 22:24:55.897485 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:56.230318 (+332833us) service_pool.cc:227] Handling call
0805 22:24:56.230678 (+ 360us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:24:56.239837 (+ 9159us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:56.239869 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:24:56.241713 (+ 1844us) log.cc:842] Serialized 26 byte log entry
0805 22:24:57.522170 (+1280457us) raft_consensus.cc:1631] Marking committed up to 3
0805 22:24:57.522214 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:57.522281 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:57.756061 (+233780us) raft_consensus.cc:1671] finished
0805 22:24:57.756080 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:57.756212 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8762,"raft.queue_time_us":38772,"raft.run_cpu_time_us":105,"raft.run_wall_time_us":106,"thread_start_us":1280080,"threads_started":1,"wal-append.queue_time_us":1512539}
I20260805 22:24:57.907287 2864 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:57.952950 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:57.972962 32589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57382 (request call id 681) took 1255 ms. Trace:
I20260805 22:24:57.966526 2877 raft_consensus.cc:2726] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:57.981946 2884 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:57.970432 31603 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:57.964327 2855 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:57.986074 2762 raft_consensus.cc:729] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:57.977317 32529 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:57.970567 2716 consensus_queue.cc:1035] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.025s
I20260805 22:24:57.975871 2656 consensus_queue.cc:1035] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260805 22:24:57.974740 31486 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:24:57.974684 31706 raft_consensus.cc:1261] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:24:58.000432 31608 raft_consensus.cc:2354] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5.
I20260805 22:24:57.984251 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:57.965363 2800 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:58.000967 2909 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:58.005206 2628 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:58.012284 2840 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:58.012140 2628 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.013059 2840 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.014626 2628 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:58.014395 2909 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.015511 32583 raft_consensus.cc:3037] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:58.016435 2840 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:58.016729 2909 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:57.966192 2876 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:58.018944 32529 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:58.020340 2876 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:58.021616 2763 raft_consensus.cc:2726] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:24:58.021921 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.022442 2763 raft_consensus.cc:2781] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:58.022843 2763 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:58.023180 2763 raft_consensus.cc:3037] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:58.012895 2633 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:58.027144 2855 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:58.014096 31486 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40740 (request call id 1239) took 2692 ms (client timeout 1583 ms). Trace:
I20260805 22:24:58.029031 2762 consensus_queue.cc:260] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:58.029691 31486 rpcz_store.cc:269] 0805 22:24:55.322041 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:55.322385 (+ 344us) service_pool.cc:227] Handling call
0805 22:24:58.013824 (+2691439us) spinlock_profiling.cc:243] Waited 2.54 s on lock 0x7b54003a2d54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:58.013950 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":5848904448}
I20260805 22:24:58.031073 2617 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:57.999236 32589 rpcz_store.cc:276] 0805 22:24:56.717750 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:56.723236 (+ 5486us) service_pool.cc:227] Handling call
0805 22:24:56.752129 (+ 28893us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:56.752333 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 7
0805 22:24:57.186220 (+433887us) log.cc:842] Serialized 14 byte log entry
0805 22:24:57.972295 (+786075us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:24:57.972412 (+ 117us) raft_consensus.cc:1631] Marking committed up to 7
0805 22:24:57.972568 (+ 156us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:57.972680 (+ 112us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:57.972866 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20013,"prepare.queue_time_us":817478,"prepare.run_cpu_time_us":8893,"prepare.run_wall_time_us":102038,"raft.queue_time_us":60656,"raft.run_cpu_time_us":278,"raft.run_wall_time_us":278,"thread_start_us":1148374,"threads_started":2,"wal-append.queue_time_us":638210}
I20260805 22:24:58.032936 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:57.985111 31713 rpcz_store.cc:276] 0805 22:24:56.204488 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:56.204632 (+ 144us) service_pool.cc:227] Handling call
0805 22:24:56.205021 (+ 389us) raft_consensus.cc:1457] Updating replica for 2 ops
0805 22:24:56.205470 (+ 449us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:24:56.205498 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0805 22:24:56.227415 (+ 21917us) log.cc:842] Serialized 52 byte log entry
0805 22:24:57.027606 (+800191us) raft_consensus.cc:1631] Marking committed up to 3
0805 22:24:57.925395 (+897789us) log.cc:842] Serialized 14 byte log entry
0805 22:24:57.940942 (+ 15547us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:57.941106 (+ 164us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:24:57.941168 (+ 62us) raft_consensus.cc:1671] finished
0805 22:24:57.941178 (+ 10us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:24:57.941421 (+ 243us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20317,"prepare.queue_time_us":314160,"prepare.run_cpu_time_us":32,"prepare.run_wall_time_us":34,"raft.queue_time_us":4990,"raft.run_cpu_time_us":483,"raft.run_wall_time_us":43655,"thread_start_us":1603208,"threads_started":2,"wal-append.queue_time_us":866686}
I20260805 22:24:58.034564 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:57.984894 2643 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.063916 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.064353 31602 raft_consensus.cc:1226] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:58.066318 2643 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:58.037698 32581 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
W20260805 22:24:58.039737 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet c5b0fd7f6d7a46b2a77e841ceda358ef, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.040553 32594 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:58.041997 31608 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:24:58.047482 2891 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:24:58.049052 32585 consensus_meta.cc:287] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.976s user 0.011s sys 0.008s
I20260805 22:24:58.053572 31597 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:58.058791 31603 raft_consensus.cc:1226] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 8. Current term is 9. Ops: []
I20260805 22:24:58.104569 31492 raft_consensus.cc:1226] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 6. Current term is 7. Ops: []
I20260805 22:24:58.105003 31594 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:24:58.063339 31604 raft_consensus.cc:1261] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260805 22:24:58.106446 31604 pending_rounds.cc:76] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c: Aborting all ops after (but not including) 5
I20260805 22:24:58.106716 31604 pending_rounds.cc:97] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260805 22:24:58.106977 31604 raft_consensus.cc:2866] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:24:58.035449 32591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57382 (request call id 685) took 1263 ms. Trace:
I20260805 22:24:58.079592 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.108719 31711 raft_consensus.cc:2379] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
W20260805 22:24:58.109614 2767 consensus_meta.cc:287] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 1.073s user 0.016s sys 0.005s
I20260805 22:24:58.109863 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:58.110832 31711 raft_consensus.cc:2379] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:58.111452 31486 raft_consensus.cc:1261] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260805 22:24:58.111670 32591 rpcz_store.cc:276] 0805 22:24:56.745905 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:56.746887 (+ 982us) service_pool.cc:227] Handling call
0805 22:24:57.998254 (+1251367us) spinlock_profiling.cc:243] Waited 306 ms on lock 0x7b5400424354. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:58.009163 (+ 10909us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":703718784}
I20260805 22:24:58.112920 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:58.113828 31714 raft_consensus.cc:1226] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260805 22:24:58.114095 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.083158 31706 raft_consensus.cc:2454] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
W20260805 22:24:58.082698 1989 consensus_meta.cc:287] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.956s user 0.013s sys 0.008s
I20260805 22:24:58.093768 32579 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:24:58.095619 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.096264 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.096724 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:58.116902 31708 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:58.120401 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.098150 31707 raft_consensus.cc:1226] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:58.121861 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.123018 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.123780 32594 raft_consensus.cc:2454] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 2.
I20260805 22:24:58.093822 2762 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:58.125334 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.126190 31709 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:24:58.126821 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:58.127468 31707 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:24:58.098850 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.129081 32591 raft_consensus.cc:2379] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 6: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:58.126739 32579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57366 (request call id 466) took 1064 ms. Trace:
W20260805 22:24:58.099882 31450 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.528s user 0.000s sys 0.009s
W20260805 22:24:58.130216 31450 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.561s user 0.001s sys 0.009s
I20260805 22:24:58.095656 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
W20260805 22:24:58.103251 31444 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.194:41897 timed out after 1.737s (SENT) [suppressed 5 similar messages]
I20260805 22:24:58.090211 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:58.069581 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet c5b0fd7f6d7a46b2a77e841ceda358ef, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.129943 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.129202 31668 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:58.099452 31704 raft_consensus.cc:1261] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260805 22:24:58.129741 32584 raft_consensus.cc:1261] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260805 22:24:58.130756 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.128389 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.129212 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.126348 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:24:58.134706 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 166297us
I20260805 22:24:58.129696 32579 rpcz_store.cc:276] 0805 22:24:57.062538 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:57.065477 (+ 2939us) service_pool.cc:227] Handling call
0805 22:24:58.126579 (+1061102us) spinlock_profiling.cc:243] Waited 978 ms on lock 0x7b5400685454. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:58.126692 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2249559808}
I20260805 22:24:58.124679 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.134194 31608 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:58.136178 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:58.137091 31668 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:58.137410 2855 raft_consensus.cc:2781] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:58.138118 31706 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:58.141197 31501 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:24:58.141746 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:58.142233 31714 raft_consensus.cc:1226] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:58.143265 2633 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:58.144930 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.116101 32583 raft_consensus.cc:2454] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:58.146129 2855 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:58.149399 2855 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:58.149667 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.150660 31706 raft_consensus.cc:2396] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260805 22:24:58.155992 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.156780 31704 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:24:58.158342 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.130131 2763 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:58.161190 31711 raft_consensus.cc:2379] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:58.160859 2633 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.163269 2763 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:58.164194 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.164969 31608 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
W20260805 22:24:58.165534 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 48941.3us
I20260805 22:24:58.167099 2633 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:58.167275 31667 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:58.160990 2628 raft_consensus.cc:2781] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:58.168242 2628 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:58.168663 2628 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:58.169128 2452 raft_consensus.cc:2741] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election decision vote started in defunct term 3: won
I20260805 22:24:58.169099 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.171110 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.158358 31708 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:58.177989 31667 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:58.147969 32585 raft_consensus.cc:2454] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:58.180220 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.184212 32586 raft_consensus.cc:1226] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:58.184401 32585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57382 (request call id 688) took 1307 ms. Trace:
I20260805 22:24:58.207578 32586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57366 (request call id 463) took 1174 ms. Trace:
I20260805 22:24:58.186239 31501 raft_consensus.cc:1261] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260805 22:24:58.207980 32586 rpcz_store.cc:276] 0805 22:24:57.033244 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:57.033711 (+ 467us) service_pool.cc:227] Handling call
0805 22:24:58.183890 (+1150179us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:24:58.206307 (+ 22417us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:24:58.207361 (+ 1054us) spinlock_profiling.cc:243] Waited 1.15 s on lock 0x7b540017dc50. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731067a09 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:24:58.207517 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2644261248}
I20260805 22:24:58.185351 32581 raft_consensus.cc:1226] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
W20260805 22:24:58.187317 31554 connection.cc:232] RPC call timeout handler was delayed by -1.414348732912913s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.22200000000000003s for a response
I20260805 22:24:58.186610 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:58.210583 31490 raft_consensus.cc:1261] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:24:58.187748 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.187433 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.185611 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:58.187000 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.214236 31554 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:58.214742 2617 consensus_queue.cc:1035] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:24:58.216269 2891 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:24:58.216883 2891 raft_consensus.cc:686] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 7 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:58.218005 2891 consensus_queue.cc:237] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.219966 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.188244 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.221589 31554 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:58.189698 32591 raft_consensus.cc:2396] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 8 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260805 22:24:58.210872 31502 raft_consensus.cc:3037] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:58.195192 31597 raft_consensus.cc:2379] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 7: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:58.194087 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.196666 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.196297 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.193696 31714 raft_consensus.cc:1226] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 6. Current term is 7. Ops: []
I20260805 22:24:58.191555 32584 raft_consensus.cc:1226] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 5. Current term is 6. Ops: [5.7-5.7]
I20260805 22:24:58.199847 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:58.226575 31594 raft_consensus.cc:2354] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5.
I20260805 22:24:58.230099 2800 consensus_queue.cc:1046] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:58.230875 31447 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:58.231972 2633 raft_consensus.cc:2781] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:58.232101 31600 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:58.232622 2633 raft_consensus.cc:686] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:58.233134 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.233896 31495 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:58.188264 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:58.234241 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.234802 31500 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:58.233780 2633 consensus_queue.cc:237] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:58.235544 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.236147 31597 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:58.236997 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.196868 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.197636 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.199103 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:58.181332 31707 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 9.
I20260805 22:24:58.193789 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.234150 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.239455 32579 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:58.237861 31492 raft_consensus.cc:2396] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 9 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260805 22:24:58.188359 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.197371 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:58.233053 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:58.198493 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.198968 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.242933 31600 raft_consensus.cc:2362] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7. Re-sending same reply.
I20260805 22:24:58.248039 31447 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:58.250710 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.249028 31706 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:58.188849 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.188681 2452 raft_consensus.cc:2741] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260805 22:24:58.207333 32585 rpcz_store.cc:276] 0805 22:24:56.876725 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:56.894098 (+ 17373us) service_pool.cc:227] Handling call
0805 22:24:58.184320 (+1290222us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:24:58.231244 2800 consensus_queue.cc:1035] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 1, Time since last communication: 0.019s
I20260805 22:24:58.254499 31486 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:24:58.252775 31494 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:58.252460 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.256675 31603 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
I20260805 22:24:58.253855 31604 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:24:58.258075 31499 raft_consensus.cc:2379] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 7: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:58.262207 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.264155 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.264977 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.265585 31600 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:24:58.191747 2855 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:58.268793 2855 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:58.269233 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.269909 31486 raft_consensus.cc:2454] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:58.274652 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.275295 31602 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:58.276381 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.277788 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.280437 31608 raft_consensus.cc:2379] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:58.277886 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.283665 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:58.287632 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.293720 31495 raft_consensus.cc:2396] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260805 22:24:58.295965 31494 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:58.282625 2728 raft_consensus.cc:2781] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:58.298758 31447 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:58.303856 31492 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:24:58.302901 31499 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:58.301746 31597 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
W20260805 22:24:58.310590 32529 connection.cc:232] RPC call timeout handler was delayed by -1.3878140677697957s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1525s for a response
W20260805 22:24:58.310901 32529 connection.cc:232] RPC call timeout handler was delayed by -1.3878140677697957s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1525s for a response
W20260805 22:24:58.311103 32529 connection.cc:232] RPC call timeout handler was delayed by -1.2519140678923577s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1676s for a response
W20260805 22:24:58.311311 32529 connection.cc:232] RPC call timeout handler was delayed by -1.0836140678729862s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18630000000000002s for a response
I20260805 22:24:58.317081 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.317454 2824 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:58.317775 31602 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:24:58.318068 2824 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:58.318498 2628 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:58.318903 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.319676 31600 raft_consensus.cc:2379] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 8: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:58.320185 2824 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:58.320521 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:58.320719 2824 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:58.268733 2877 consensus_queue.cc:1046] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.043s
I20260805 22:24:58.321448 31609 raft_consensus.cc:2379] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:58.321599 2876 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:58.322321 2876 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.323344 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:58.323287 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:58.324151 31594 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:58.324546 2876 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:58.321770 2631 consensus_queue.cc:1046] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.005s
I20260805 22:24:58.326798 31554 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:58.327373 2617 raft_consensus.cc:3032] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:58.327769 2617 raft_consensus.cc:729] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:58.328649 2716 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:24:58.328994 31447 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:58.328801 2617 consensus_queue.cc:260] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:58.324059 32585 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:58.329890 31596 raft_consensus.cc:1226] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
I20260805 22:24:58.309999 2840 raft_consensus.cc:2781] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:58.325135 2876 raft_consensus.cc:3032] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:58.332844 2617 raft_consensus.cc:3037] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:58.333330 2716 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:58.333726 2716 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:24:58.334510 2800 consensus_queue.cc:1035] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20260805 22:24:58.337944 31554 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:58.298759 2728 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:58.338981 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.340072 2867 raft_consensus.cc:3032] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:24:58.340348 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:58.341954 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:58.344014 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:58.331543 2824 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:58.347786 31499 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:24:58.320617 2628 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:58.351904 32529 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:58.361362 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.332858 2840 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:58.359131 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.363058 31495 raft_consensus.cc:1226] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:58.343320 2867 raft_consensus.cc:729] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:58.345652 31594 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:24:58.339051 2876 raft_consensus.cc:729] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:58.363941 31494 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:24:58.342861 2728 raft_consensus.cc:3037] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:58.367055 2840 raft_consensus.cc:3037] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:58.374356 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.409783 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:58.369161 2884 raft_consensus.cc:2781] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:58.416296 31594 raft_consensus.cc:2379] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:58.419999 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.372979 2763 raft_consensus.cc:2726] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260805 22:24:58.424497 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 83281.2us
I20260805 22:24:58.425829 31500 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:24:58.426893 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.421964 31596 raft_consensus.cc:1226] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:58.345142 32587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:60186
I20260805 22:24:58.430712 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.439653 31500 raft_consensus.cc:1226] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 6. Current term is 7. Ops: []
I20260805 22:24:58.420985 2884 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:58.420497 2869 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:58.433689 31554 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:58.443809 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.453035 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.453907 31594 raft_consensus.cc:2379] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 4: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:58.459920 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.460716 31596 raft_consensus.cc:2379] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:58.444038 2884 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:58.462580 2842 raft_consensus.cc:2781] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election won for term 4
W20260805 22:24:58.470024 2926 log.cc:924] Time spent T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.066s user 0.001s sys 0.000s
I20260805 22:24:58.460301 2869 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.471395 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:24:58.482043 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 63335.3us
I20260805 22:24:58.457203 2670 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:58.478669 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.522300 31594 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:24:58.499382 2894 raft_consensus.cc:2781] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election won for term 3
I20260805 22:24:58.481284 31601 raft_consensus.cc:2454] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:58.514920 32587 raft_consensus.cc:595] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:58.470145 2824 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:58.473778 2909 consensus_queue.cc:1046] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 4.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.011s
I20260805 22:24:58.474701 2842 raft_consensus.cc:686] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:58.525848 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.526844 32587 raft_consensus.cc:603] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:24:58.536844 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.514218 2891 raft_consensus.cc:2781] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:58.533691 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.532730 31554 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:58.542629 2842 consensus_queue.cc:237] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:58.546408 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.547350 31596 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:58.547636 31502 raft_consensus.cc:2454] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
W20260805 22:24:58.551036 2927 log.cc:924] Time spent T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.091s user 0.000s sys 0.000s
I20260805 22:24:58.554721 2914 raft_consensus.cc:3032] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:58.536486 2670 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:58.555761 2633 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:58.555996 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.558583 2670 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:58.559288 32529 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:58.560541 32529 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:58.562433 2763 raft_consensus.cc:2726] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:24:58.560631 2670 raft_consensus.cc:2726] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:58.549186 31359 catalog_manager.cc:5495] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 3 to 6, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 6 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:58.577589 2869 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:58.476527 2867 consensus_queue.cc:260] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.578786 2867 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:58.509553 2876 consensus_queue.cc:260] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.576608 31359 catalog_manager.cc:5495] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 4 to 5, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 5 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:24:58.580544 2876 raft_consensus.cc:3037] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:58.547297 31597 raft_consensus.cc:2379] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 9: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:58.568683 31357 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34400 (request call id 285) took 9891 ms. Trace:
I20260805 22:24:58.582579 31359 catalog_manager.cc:5495] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 2 to 3, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:24:58.588653 32579 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
W20260805 22:24:58.595101 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 89920.4us
I20260805 22:24:58.596184 31554 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:58.597544 32529 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:58.555133 2914 raft_consensus.cc:729] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:58.600724 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.604409 2914 consensus_queue.cc:260] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.608590 31668 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:58.608068 31604 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 9.
I20260805 22:24:58.607478 2914 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:58.599010 2891 raft_consensus.cc:686] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:58.612172 2891 consensus_queue.cc:237] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 4.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.613662 31600 raft_consensus.cc:1226] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 8. Current term is 9. Ops: []
I20260805 22:24:58.599215 2824 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election won for term 9
I20260805 22:24:58.620019 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.620687 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:58.620915 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.602887 2869 consensus_queue.cc:1035] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:24:58.653715 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.655009 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.655033 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.655857 31713 raft_consensus.cc:2396] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260805 22:24:58.656318 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.657032 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.657500 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.657641 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.658130 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.658290 31705 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:58.658643 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.659140 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.659260 31715 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:24:58.659391 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.659758 31709 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:24:58.653594 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.660223 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.660794 31715 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:24:58.661851 31708 raft_consensus.cc:1261] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260805 22:24:58.662987 32529 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:58.656595 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:58.660037 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.664866 31707 raft_consensus.cc:2354] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5.
I20260805 22:24:58.659155 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.607977 2670 raft_consensus.cc:2726] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:58.599408 2633 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.615495 2800 raft_consensus.cc:2781] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:24:58.665426 2617 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:58.648602 2631 raft_consensus.cc:2781] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:24:58.672197 2728 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:58.662885 2934 log.cc:924] Time spent T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.122s user 0.000s sys 0.001s
W20260805 22:24:58.667357 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 50465us
I20260805 22:24:58.675940 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:58.676738 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.663862 31359 catalog_manager.cc:5495] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 3 to 6, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 6 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:58.603479 2864 consensus_queue.cc:1046] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260805 22:24:58.631296 32585 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:24:58.685709 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.686318 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:58.686498 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.687057 2617 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.688285 31494 raft_consensus.cc:1226] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 6. Current term is 7. Ops: [6.4-6.4]
W20260805 22:24:58.688395 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 45cd56d4e96040f98dbf1ae001a1005a, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.688755 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.678375 2824 raft_consensus.cc:686] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 9 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:58.626613 2628 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:58.690383 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:58.690739 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.714233 31708 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:24:58.678580 2800 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:58.669245 2445 raft_consensus.cc:2726] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:58.694744 31490 raft_consensus.cc:2379] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:58.694784 31500 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:24:58.721719 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.696972 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:24:58.703024 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 45cd56d4e96040f98dbf1ae001a1005a, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.705899 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.706125 31609 raft_consensus.cc:2379] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 3: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:58.714303 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.716768 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.717097 32529 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:58.720678 31668 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:58.723274 31502 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:24:58.695665 2631 raft_consensus.cc:686] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:58.727000 2824 consensus_queue.cc:237] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.724628 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.727150 2631 consensus_queue.cc:237] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:58.724718 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.722795 31711 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:58.737649 31714 raft_consensus.cc:2396] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260805 22:24:58.738440 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.737623 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.741868 31704 raft_consensus.cc:2379] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 6: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:58.720443 2800 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:58.748065 31706 raft_consensus.cc:1261] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
I20260805 22:24:58.748016 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.744575 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.749320 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.754029 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:58.743629 2445 raft_consensus.cc:2726] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:58.756424 31714 raft_consensus.cc:1261] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260805 22:24:58.728250 2628 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:58.759433 31486 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:24:58.695855 2884 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.761286 2869 raft_consensus.cc:3032] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 8 LEADER]: Stepping down as leader of term 8
I20260805 22:24:58.764756 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.698238 31359 catalog_manager.cc:5495] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
W20260805 22:24:58.762691 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 64801.4us
I20260805 22:24:58.735960 2855 raft_consensus.cc:2781] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:24:58.764187 31600 raft_consensus.cc:1226] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 6. Current term is 7. Ops: []
I20260805 22:24:58.764580 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:58.765638 31502 raft_consensus.cc:2396] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260805 22:24:58.761206 31500 raft_consensus.cc:1226] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 8. Current term is 9. Ops: []
I20260805 22:24:58.768347 31494 raft_consensus.cc:2379] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 8: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:58.764072 31499 raft_consensus.cc:2454] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:24:58.771512 31494 raft_consensus.cc:1261] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260805 22:24:58.774619 31499 raft_consensus.cc:1261] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260805 22:24:58.775115 31708 raft_consensus.cc:2379] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 8: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:58.588176 31357 rpcz_store.cc:276] 0805 22:24:48.682505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:48.682680 (+ 175us) service_pool.cc:227] Handling call
0805 22:24:58.568414 (+9885734us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0805 22:24:55.710855 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0805 22:24:55.711386 (+ 531us) write_op.cc:432] Acquiring schema lock in shared mode
0805 22:24:55.711424 (+ 38us) write_op.cc:435] Acquired schema lock
0805 22:24:55.711436 (+ 12us) tablet.cc:660] Decoding operations
0805 22:24:55.712628 (+ 1192us) write_op.cc:620] Acquiring the partition lock for write op
0805 22:24:55.712699 (+ 71us) write_op.cc:641] Partition lock acquired for write op
0805 22:24:55.712729 (+ 30us) tablet.cc:683] Acquiring locks for 9 operations
0805 22:24:55.806362 (+ 93633us) tablet.cc:699] Row locks acquired
0805 22:24:55.809561 (+ 3199us) write_op.cc:260] PREPARE: finished
0805 22:24:55.891731 (+ 82170us) write_op.cc:270] Start()
0805 22:24:55.911043 (+ 19312us) write_op.cc:276] Timestamp: P: 1785968695888629 usec, L: 0
0805 22:24:55.914417 (+ 3374us) op_driver.cc:331] REPLICATION: starting
0805 22:24:55.986239 (+ 71822us) log.cc:842] Serialized 3627 byte log entry
0805 22:24:57.918593 (+1932354us) op_driver.cc:444] REPLICATION: finished
0805 22:24:58.101680 (+183087us) write_op.cc:301] APPLY: starting
0805 22:24:58.101825 (+ 145us) tablet.cc:1365] starting BulkCheckPresence
0805 22:24:58.102142 (+ 317us) tablet.cc:1368] finished BulkCheckPresence
0805 22:24:58.102274 (+ 132us) tablet.cc:1370] starting ApplyRowOperation cycle
0805 22:24:58.105160 (+ 2886us) tablet.cc:1381] finished ApplyRowOperation cycle
0805 22:24:58.105534 (+ 374us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0805 22:24:58.105594 (+ 60us) write_op.cc:312] APPLY: finished
0805 22:24:58.106104 (+ 510us) log.cc:842] Serialized 71 byte log entry
0805 22:24:58.106315 (+ 211us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:58.106570 (+ 255us) write_op.cc:454] Released schema lock
0805 22:24:58.106629 (+ 59us) write_op.cc:341] FINISH: Updating metrics
0805 22:24:58.107013 (+ 384us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:58.107067 (+ 54us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":322279,"reported_tablets":33,"child_traces":[["op",{"apply.queue_time_us":182692,"apply.run_cpu_time_us":4403,"apply.run_wall_time_us":5745,"num_ops":9,"prepare.queue_time_us":396863,"prepare.run_cpu_time_us":49381,"prepare.run_wall_time_us":1150208,"raft.queue_time_us":179428,"raft.run_cpu_time_us":53530,"raft.run_wall_time_us":81440,"replication_time_us":2003897,"thread_start_us":1329352,"threads_started":4,"wal-append.queue_time_us":1046987}]]}
I20260805 22:24:58.782711 31609 raft_consensus.cc:1226] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:58.788240 31594 raft_consensus.cc:2379] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 6: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:58.791652 31704 raft_consensus.cc:1261] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 3. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20260805 22:24:58.790656 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:58.764952 2869 raft_consensus.cc:729] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:58.799113 31711 raft_consensus.cc:2354] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5.
I20260805 22:24:58.802132 31492 raft_consensus.cc:1226] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4. Ops: []
I20260805 22:24:58.804010 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.805186 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.805060 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:58.805881 31707 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:24:58.805999 31609 raft_consensus.cc:2379] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 4: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:58.806699 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.807642 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.805527 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.805130 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.810375 31714 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:24:58.813781 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.817142 31444 leader_election.cc:398] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5.
I20260805 22:24:58.817298 31708 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:24:58.818701 31444 leader_election.cc:401] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260805 22:24:58.822152 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.825345 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:58.826149 31609 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 7
I20260805 22:24:58.828039 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.829636 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.822841 31711 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:24:58.810753 2894 raft_consensus.cc:686] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
W20260805 22:24:58.841158 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 07310455e35e48f8ba4eb9b72a69113b, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:58.841594 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 07310455e35e48f8ba4eb9b72a69113b, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.842482 2445 consensus_queue.cc:1035] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:24:58.843155 2617 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:58.808687 2716 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.849785 31495 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:24:58.846097 31444 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:58.812763 2840 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:58.851866 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.817566 1989 consensus_queue.cc:1046] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.031s
I20260805 22:24:58.861754 1989 consensus_queue.cc:1225] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer f8e9f4c756da4203917c464e2b179994 log is divergent from this leader: its last log entry 5.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
I20260805 22:24:58.862360 1989 raft_consensus.cc:3032] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:24:58.862685 1989 raft_consensus.cc:729] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:58.863070 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.863653 1989 consensus_queue.cc:260] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:58.864675 31554 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:58.864988 1989 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:58.865242 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.865639 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:58.866513 2840 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:58.870041 2628 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:58.870822 2716 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:58.873346 2877 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:24:58.849340 2633 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:58.870013 2894 consensus_queue.cc:237] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:58.878700 2948 consensus_queue.cc:1035] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:24:58.879362 31554 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:58.873924 31556 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:58.880918 2948 raft_consensus.cc:2781] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:58.881369 2948 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:58.881191 2913 raft_consensus.cc:2726] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:58.881801 2948 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:58.882506 31554 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:58.883606 2797 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:58.880277 2731 consensus_queue.cc:1035] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.004s
W20260805 22:24:58.880210 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 10de517cc0a24f20b5885091e99e252d, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:58.886976 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 10de517cc0a24f20b5885091e99e252d, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.882452 31556 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:58.859862 31359 catalog_manager.cc:5495] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 1 to 4, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 4 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:24:58.865983 2728 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:58.884052 2797 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:58.851686 2945 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5.
I20260805 22:24:58.870714 2884 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:58.851531 2938 log.cc:924] Time spent T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.100s user 0.001s sys 0.001s
I20260805 22:24:58.903733 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.904908 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.908154 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.908450 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.894882 31600 raft_consensus.cc:2379] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 3: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:58.905611 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.915021 32579 raft_consensus.cc:2396] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260805 22:24:58.869227 31609 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:24:58.904548 31359 catalog_manager.cc:5495] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c reported cstate change: term changed from 3 to 5, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 5 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:24:58.907074 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.890161 31713 raft_consensus.cc:2454] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
W20260805 22:24:58.922727 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 65374.3us
I20260805 22:24:58.907807 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.912700 32589 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
W20260805 22:24:58.933066 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 45308.5us
I20260805 22:24:58.934471 31554 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:58.941043 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:24:58.943868 32579 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:24:58.851442 2855 raft_consensus.cc:686] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 7 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:58.899183 2869 consensus_queue.cc:260] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.953603 2728 consensus_queue.cc:1046] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:24:58.954298 2869 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:24:58.954770 2728 raft_consensus.cc:3032] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:58.955211 2728 raft_consensus.cc:729] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:58.956303 31556 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:58.956123 2728 consensus_queue.cc:260] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:58.946275 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:58.894793 2797 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:58.895382 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:58.931300 2948 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:58.946879 2824 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:58.952759 2864 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:58.873935 2877 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:58.950291 31554 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:58.965898 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:58.906402 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:24:58.967527 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 121483us
I20260805 22:24:58.929188 2731 raft_consensus.cc:2726] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:58.969169 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:58.993635 31668 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:58.958014 2884 raft_consensus.cc:2726] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:24:58.969645 32583 raft_consensus.cc:1226] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:58.973152 2877 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:58.969261 2824 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:59.009663 2824 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:58.900820 2800 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.014875 2800 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:58.976667 2948 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:58.957355 2728 raft_consensus.cc:3037] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:58.976500 31444 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:58.986573 2767 consensus_queue.cc:1046] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:24:58.987550 31556 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:58.988131 2876 raft_consensus.cc:2781] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:24:58.988633 2891 consensus_queue.cc:1046] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260805 22:24:58.989470 32529 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:24:58.986156 2855 consensus_queue.cc:237] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:58.993813 31705 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 7.
W20260805 22:24:59.030248 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 194615us
I20260805 22:24:58.971028 2864 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:58.994771 31554 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.032369 31444 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf
W20260805 22:24:59.035002 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 45664.5us
W20260805 22:24:59.035636 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 48185.1us
I20260805 22:24:59.025336 2633 raft_consensus.cc:2781] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:59.042452 31556 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:59.045915 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 51310.9us
I20260805 22:24:59.046324 2955 raft_consensus.cc:2781] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:24:59.047592 2948 raft_consensus.cc:3032] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:24:59.048029 2948 raft_consensus.cc:729] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:59.048198 2955 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:59.031629 2670 raft_consensus.cc:2781] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:59.031350 2864 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:59.050750 2955 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:59.036465 2876 raft_consensus.cc:686] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [term 7 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:59.040306 2913 raft_consensus.cc:2741] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260805 22:24:59.060041 31667 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:59.042100 2633 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:59.054564 2840 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:24:59.058404 2867 consensus_queue.cc:1035] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.007s
I20260805 22:24:59.062980 2953 raft_consensus.cc:2781] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:59.056321 2445 raft_consensus.cc:3032] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:24:59.079927 2445 raft_consensus.cc:729] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:59.080830 2445 consensus_queue.cc:260] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:59.082242 2445 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:59.065647 2800 raft_consensus.cc:2726] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:24:59.072956 2633 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:59.054385 2884 consensus_queue.cc:1035] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.031s
I20260805 22:24:59.067090 31444 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:59.084750 2877 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.086599 31444 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:59.087493 31444 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:59.088171 2643 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:24:59.088548 2643 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:59.088771 31444 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:59.089078 2842 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:24:59.051260 2670 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:59.089954 2670 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:59.090639 32583 raft_consensus.cc:1261] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260805 22:24:59.059377 2876 consensus_queue.cc:237] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.094341 2643 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:24:59.052892 2731 raft_consensus.cc:2781] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:59.079484 2953 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:59.089534 2842 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:59.101753 2953 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:59.115396 2842 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:59.102988 31444 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:24:59.103768 2941 raft_consensus.cc:2781] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:59.116851 2941 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:59.117323 2941 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:59.118978 2945 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader election won for term 9
I20260805 22:24:59.119519 2945 raft_consensus.cc:686] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 9 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:59.120465 2945 consensus_queue.cc:237] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.126519 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:59.087878 2877 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:59.089574 2909 raft_consensus.cc:2781] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:24:59.089790 2948 consensus_queue.cc:260] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:59.105691 2944 consensus_queue.cc:1035] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:24:59.139184 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:59.137363 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:59.139186 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
W20260805 22:24:59.133399 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 91581us
I20260805 22:24:59.141848 31608 raft_consensus.cc:1226] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 9 LEADER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 8. Current term is 9. Ops: []
I20260805 22:24:59.100888 2840 raft_consensus.cc:686] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:24:59.145495 32529 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.126561 2731 raft_consensus.cc:686] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:59.148213 31667 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:59.150595 2824 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.123988 2797 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:59.153452 2797 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:59.153941 2767 raft_consensus.cc:2726] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:59.155786 31603 raft_consensus.cc:1261] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260805 22:24:59.157982 2633 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.160426 2633 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:59.161757 31600 raft_consensus.cc:2379] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 7: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:24:59.135197 2909 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:24:59.164533 2909 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
W20260805 22:24:59.185565 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 48313.6us
I20260805 22:24:59.186429 31667 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:59.197674 2762 raft_consensus.cc:2781] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:24:59.200120 2762 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:59.200485 2762 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
W20260805 22:24:59.198017 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 52626.6us
I20260805 22:24:59.210989 2762 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:59.198191 2631 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election won for term 8
I20260805 22:24:59.213867 2631 raft_consensus.cc:686] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:59.214845 2631 consensus_queue.cc:237] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.219358 2762 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:59.153034 2731 consensus_queue.cc:237] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.2, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.223423 2643 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.224647 2941 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.200505 2840 consensus_queue.cc:237] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:59.226382 2824 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:59.228268 2953 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:59.230378 31667 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:59.245589 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 115839us
I20260805 22:24:59.230963 2948 raft_consensus.cc:3037] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:59.259773 2953 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:59.259649 2956 raft_consensus.cc:2726] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:59.265662 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:59.266781 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:59.267431 32583 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:59.254566 2643 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:59.272416 32529 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:59.247076 2842 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:59.275321 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 226986us
I20260805 22:24:59.271407 31444 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.279039 2975 raft_consensus.cc:2781] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:24:59.279464 2975 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:59.279811 2975 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:59.275775 2864 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:59.282397 2864 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:59.278409 2670 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:59.285405 32585 raft_consensus.cc:1261] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260805 22:24:59.286972 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:59.226423 2955 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:59.289577 31715 raft_consensus.cc:1226] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 5. Current term is 6. Ops: []
I20260805 22:24:59.289690 31556 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:59.290342 2941 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:59.291036 32583 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:24:59.298419 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:59.258164 2909 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:59.318251 2978 log.cc:924] Time spent T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260805 22:24:59.319121 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:59.309312 31556 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:59.309538 31667 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:59.312990 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:59.359298 31492 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 6
W20260805 22:24:59.358747 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 96573.2us
I20260805 22:24:59.359036 31501 raft_consensus.cc:2379] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 9: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
W20260805 22:24:59.359586 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 50356.9us
I20260805 22:24:59.368769 31705 raft_consensus.cc:1261] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260805 22:24:59.370064 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:59.370673 31713 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 9
I20260805 22:24:59.374320 2909 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:59.375680 2955 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:59.377895 31556 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.380543 31444 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.367203 2891 raft_consensus.cc:2781] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:24:59.378865 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:59.386574 31715 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:59.391636 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:59.363982 2941 raft_consensus.cc:2726] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260805 22:24:59.396399 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 96099us
I20260805 22:24:59.394407 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:59.401719 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:59.402120 31711 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 8
I20260805 22:24:59.396478 2971 raft_consensus.cc:2726] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:59.368528 2884 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:24:59.385244 2956 raft_consensus.cc:2726] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:59.406309 31713 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 9.
I20260805 22:24:59.409806 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:59.412711 31667 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:59.414583 31556 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:59.415258 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:59.408912 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:59.417500 2953 consensus_queue.cc:1046] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:59.389123 2891 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:59.397195 2670 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:59.424226 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 56519.7us
I20260805 22:24:59.418833 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:59.426400 31500 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:59.427737 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:59.433532 2891 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:59.433894 2956 raft_consensus.cc:2726] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:59.413213 2797 raft_consensus.cc:2726] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:59.440963 31444 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.441808 31556 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:59.413913 2909 consensus_queue.cc:1035] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
W20260805 22:24:59.539088 2989 log.cc:924] Time spent T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.109s user 0.000s sys 0.001s
I20260805 22:24:59.455538 31667 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:24:59.559556 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 104100us
I20260805 22:24:59.444777 2842 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:59.560387 31667 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:59.509913 2975 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:59.561947 32583 raft_consensus.cc:2379] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 8: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:24:59.562789 2975 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:24:59.513720 2997 log.cc:924] Time spent T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20260805 22:24:59.533713 31492 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:59.517761 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
W20260805 22:24:59.564080 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 122373us
I20260805 22:24:59.564821 31494 raft_consensus.cc:2379] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:24:59.476001 31711 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 8.
I20260805 22:24:59.471103 2953 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:24:59.459594 2988 log.cc:924] Time spent T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.173s user 0.000s sys 0.001s
I20260805 22:24:59.451531 2884 consensus_queue.cc:1035] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.115s
I20260805 22:24:59.455729 2955 raft_consensus.cc:3032] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:24:59.456714 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:59.451619 2824 consensus_queue.cc:1035] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.116s
I20260805 22:24:59.569057 2824 raft_consensus.cc:2726] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:24:59.451368 2985 raft_consensus.cc:2726] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:24:59.468863 31715 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:24:59.569443 2891 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:59.570120 2855 raft_consensus.cc:2741] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260805 22:24:59.570381 2953 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.571398 2855 raft_consensus.cc:2781] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:24:59.571765 2855 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:59.572105 2855 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:59.467707 31500 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 8.
I20260805 22:24:59.567463 2955 raft_consensus.cc:729] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:59.565490 31492 raft_consensus.cc:1226] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:59.563627 31667 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:59.565006 31556 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.578898 31556 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:59.579653 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 128699us
W20260805 22:24:59.589915 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 149066us
I20260805 22:24:59.592320 31554 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.579898 2909 raft_consensus.cc:2726] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:59.599721 32581 raft_consensus.cc:1226] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6. Ops: []
I20260805 22:24:59.601795 32579 raft_consensus.cc:1261] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260805 22:24:59.602861 2643 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:59.603364 2643 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.583909 2797 raft_consensus.cc:2726] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:59.605233 2643 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:59.609006 2855 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:59.584080 2955 consensus_queue.cc:260] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.610733 2955 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:24:59.617774 31444 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.573909 2891 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:59.619136 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:59.619475 2953 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:59.619616 2452 consensus_queue.cc:1035] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260805 22:24:59.620312 32587 raft_consensus.cc:2379] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 6: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
W20260805 22:24:59.622726 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 59171.3us
I20260805 22:24:59.626133 31667 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:59.627028 31667 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:59.611121 2855 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:59.617072 2985 raft_consensus.cc:2726] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:59.614626 2867 raft_consensus.cc:2726] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:24:59.616974 2956 raft_consensus.cc:2726] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:59.630059 2452 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:24:59.639156 31554 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.631573 2633 raft_consensus.cc:2726] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260805 22:24:59.639940 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 49106.1us
I20260805 22:24:59.642030 31444 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:59.644493 2840 raft_consensus.cc:2781] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:24:59.644891 2840 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:59.645252 2840 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:59.634966 2884 consensus_queue.cc:1035] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
I20260805 22:24:59.650409 31554 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:24:59.651044 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 264926us
I20260805 22:24:59.637957 2855 raft_consensus.cc:2781] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:24:59.656016 31554 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:59.638847 2452 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:59.667011 31554 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:59.668311 32587 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:60186
I20260805 22:24:59.669000 32587 raft_consensus.cc:595] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.669497 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:24:59.670405 31594 raft_consensus.cc:2396] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf for term 6 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260805 22:24:59.634522 2869 consensus_queue.cc:1046] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.022s
I20260805 22:24:59.671984 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:24:59.681963 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:59.657393 2855 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:24:59.649683 2867 raft_consensus.cc:2726] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:59.692649 31603 raft_consensus.cc:1226] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 5. Current term is 6. Ops: []
I20260805 22:24:59.681927 31444 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:24:59.689872 31609 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 7.
I20260805 22:24:59.693673 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:59.694311 32594 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:24:59.691913 2855 raft_consensus.cc:3037] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:24:59.696851 31493 raft_consensus.cc:1261] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260805 22:24:59.697971 31490 raft_consensus.cc:1226] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 5. Current term is 6. Ops: []
I20260805 22:24:59.667948 2797 raft_consensus.cc:2781] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election won for term 8
I20260805 22:24:59.701964 2797 raft_consensus.cc:686] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 8 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:59.702817 2797 consensus_queue.cc:237] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.666836 2948 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election won for term 9
I20260805 22:24:59.705355 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:59.705740 2948 raft_consensus.cc:686] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 9 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:59.706182 31709 raft_consensus.cc:2379] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 8: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:24:59.691071 2953 consensus_queue.cc:1046] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260805 22:24:59.707290 2953 raft_consensus.cc:3032] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 LEADER]: Stepping down as leader of term 8
I20260805 22:24:59.707638 2953 raft_consensus.cc:729] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:24:59.708447 2953 consensus_queue.cc:260] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.689296 2913 raft_consensus.cc:2781] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:24:59.709825 2913 raft_consensus.cc:686] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 6 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:59.710705 2913 consensus_queue.cc:237] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.711172 3008 consensus_queue.cc:1046] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:24:59.657177 2824 consensus_queue.cc:1035] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.055s
I20260805 22:24:59.712172 3008 raft_consensus.cc:3032] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:24:59.712540 3008 raft_consensus.cc:729] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:59.667846 2452 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:59.713274 3008 consensus_queue.cc:260] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:59.714948 3008 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:24:59.692101 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:24:59.692800 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:59.717103 31600 raft_consensus.cc:1226] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5. Ops: []
I20260805 22:24:59.718585 2948 consensus_queue.cc:237] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.720151 31601 raft_consensus.cc:1226] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:24:59.694924 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:24:59.723500 31667 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:24:59.737075 31554 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.739349 2840 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:59.741665 2840 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:24:59.724493 2631 raft_consensus.cc:2741] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260805 22:24:59.747617 32529 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:59.748574 2975 raft_consensus.cc:2726] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:59.749007 32529 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:59.749845 2975 raft_consensus.cc:2726] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:59.750216 32529 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:59.752394 2975 raft_consensus.cc:2726] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:59.752789 32529 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:59.753566 2975 raft_consensus.cc:2726] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:24:59.754886 31667 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:59.756596 31554 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.757158 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:24:59.758983 2953 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:24:59.761312 2955 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:24:59.767417 31501 raft_consensus.cc:1261] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260805 22:24:59.768536 32529 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:59.769899 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 88086.8us
I20260805 22:24:59.771236 31554 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.759476 2633 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:24:59.762398 2728 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:24:59.762450 2824 raft_consensus.cc:2726] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260805 22:24:59.789661 31444 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.794093 32529 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:59.783313 2633 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:24:59.805538 32594 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:24:59.805825 2633 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:59.782290 2884 raft_consensus.cc:2726] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260805 22:24:59.778568 2867 consensus_queue.cc:1046] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260805 22:24:59.793273 3007 raft_consensus.cc:2726] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:24:59.834420 31486 raft_consensus.cc:1226] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 3. Current term is 4. Ops: []
W20260805 22:24:59.837895 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 58887.4us
I20260805 22:24:59.838701 31444 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:59.836540 31556 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
W20260805 22:24:59.843250 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 88407.5us
I20260805 22:24:59.828085 2909 raft_consensus.cc:2726] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:24:59.845892 3007 consensus_queue.cc:1046] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 5.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.020s
W20260805 22:24:59.890103 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 48611.9us
I20260805 22:24:59.828056 3008 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:24:59.893395 31667 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
W20260805 22:24:59.893512 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 51355.8us
I20260805 22:24:59.915757 32529 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:24:59.917354 3015 log.cc:924] Time spent T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260805 22:24:59.918910 3010 raft_consensus.cc:3032] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:24:59.919338 3010 raft_consensus.cc:729] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:59.920176 3010 consensus_queue.cc:260] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:24:59.918753 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 74505.4us
I20260805 22:24:59.922880 31667 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:59.924201 31667 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:59.924398 2975 consensus_queue.cc:1046] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:24:59.925714 2956 raft_consensus.cc:2726] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:24:59.925836 2993 raft_consensus.cc:3032] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:24:59.926227 2993 raft_consensus.cc:729] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:59.926412 2956 raft_consensus.cc:2726] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:24:59.927089 2993 consensus_queue.cc:260] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:59.928459 31667 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:59.931289 2762 raft_consensus.cc:2726] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:24:59.932037 2762 raft_consensus.cc:2726] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260805 22:24:59.932912 3018 log.cc:924] Time spent T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.149s user 0.002s sys 0.002s
I20260805 22:24:59.933079 32594 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
mode: GRACEFUL
new_leader_uuid: "f8e9f4c756da4203917c464e2b179994"
from {username='slave'} at 127.0.0.1:60186
I20260805 22:24:59.933771 32594 raft_consensus.cc:595] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Received request to transfer leadership to TS f8e9f4c756da4203917c464e2b179994
I20260805 22:24:59.934092 32594 raft_consensus.cc:603] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:24:59.936075 2891 raft_consensus.cc:3032] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:24:59.936517 2891 raft_consensus.cc:729] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:59.936789 3010 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
W20260805 22:24:59.915608 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 79154.4us
I20260805 22:24:59.938319 31556 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:24:59.941147 32594 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:60186
W20260805 22:24:59.894537 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 55941.7us
I20260805 22:24:59.941938 32594 raft_consensus.cc:595] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.921566 2767 raft_consensus.cc:2726] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:24:59.911314 3032 raft_consensus.cc:2726] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:24:59.939195 2985 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:24:59.948307 31667 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:24:59.895150 2840 raft_consensus.cc:2781] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:24:59.955058 31444 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.913314 2452 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:24:59.959331 3023 log.cc:924] Time spent T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.187s user 0.003s sys 0.000s
I20260805 22:24:59.953768 2985 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.961243 2985 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:59.961839 2767 raft_consensus.cc:979] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf: : Instructing follower feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:59.962334 2767 raft_consensus.cc:1067] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Signalling peer feb5e7ab4b1541beaf2674c3c5500385 to start an election
I20260805 22:24:59.965354 32594 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:60186
I20260805 22:24:59.956425 2840 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:24:59.966008 32594 raft_consensus.cc:595] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.967373 32587 raft_consensus.cc:1261] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260805 22:24:59.968135 32594 raft_consensus.cc:603] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:24:59.958009 3032 raft_consensus.cc:2781] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:24:59.969576 3032 raft_consensus.cc:686] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:24:59.970536 3032 consensus_queue.cc:237] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:59.969084 31713 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:58348
I20260805 22:24:59.971906 31713 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:24:59.972239 31713 raft_consensus.cc:3037] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:59.953711 2869 consensus_queue.cc:1035] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 3, Time since last communication: 0.019s
W20260805 22:24:59.967978 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 187209us
I20260805 22:24:59.963415 2643 raft_consensus.cc:2726] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:24:59.978667 32529 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:24:59.985961 2716 consensus_queue.cc:1035] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:24:59.986797 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260805 22:24:59.973647 2840 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:24:59.990072 31486 raft_consensus.cc:1261] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260805 22:24:59.990806 2993 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:24:59.990976 2975 raft_consensus.cc:2726] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:24:59.992664 2728 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:24:59.993218 2728 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.956187 2891 consensus_queue.cc:260] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:24:59.994603 2452 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:24:59.995268 2728 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:24:59.993249 31714 raft_consensus.cc:1261] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260805 22:24:59.995908 31667 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:24:59.975515 2944 consensus_queue.cc:1035] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.022s
I20260805 22:24:59.965742 2633 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:24:59.994897 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:24:59.993932 31708 raft_consensus.cc:1261] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 4 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260805 22:24:59.997793 31501 raft_consensus.cc:1261] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260805 22:24:59.982805 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:00.001657 31704 raft_consensus.cc:1261] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 4. (term mismatch)
I20260805 22:25:00.024091 31704 pending_rounds.cc:76] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385: Aborting all ops after (but not including) 3
I20260805 22:25:00.024360 31704 pending_rounds.cc:97] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385: Aborting uncommitted NO_OP operation due to leader change: 6.4
I20260805 22:25:00.015364 31713 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.017886 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:00.027073 31601 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:00.022771 31707 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:25:00.023919 31495 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:25:00.040446 31444 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:00.024631 31704 raft_consensus.cc:2866] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:25:00.069809 31713 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:00.070104 31601 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 8.
I20260805 22:25:00.072180 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:00.072891 32587 raft_consensus.cc:3032] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:25:00.074573 31502 raft_consensus.cc:1226] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 7. Current term is 8. Ops: []
I20260805 22:25:00.073295 32587 raft_consensus.cc:729] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:24:59.975342 2985 consensus_queue.cc:1035] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.071s
I20260805 22:25:00.078239 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:24:59.995741 3026 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:25:00.048761 2452 raft_consensus.cc:2781] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:25:00.081171 31492 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:25:00.078204 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:00.083633 31709 raft_consensus.cc:2379] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:00.035698 2944 consensus_queue.cc:1225] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer 6bf8c694a0ba40978195ae044a917caf log is divergent from this leader: its last log entry 6.7 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 6
W20260805 22:25:00.088363 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 90699.1us
I20260805 22:25:00.091043 31444 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:00.088634 31715 raft_consensus.cc:1261] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 2. Preceding OpId from leader: term: 5 index: 3. (index mismatch)
I20260805 22:25:00.069597 2842 raft_consensus.cc:3032] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:25:00.101913 32587 consensus_queue.cc:260] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.114885 32587 raft_consensus.cc:3037] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:00.071537 2716 consensus_queue.cc:1035] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.048s
I20260805 22:25:00.083617 2452 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
W20260805 22:25:00.122243 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 72989.9us
I20260805 22:25:00.121459 2452 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:00.121583 2941 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:00.126345 2941 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.128595 2941 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:00.080986 2767 consensus_queue.cc:1046] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.035s
W20260805 22:25:00.130776 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 154604us
I20260805 22:25:00.103849 2842 raft_consensus.cc:729] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:00.115978 2643 raft_consensus.cc:2726] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:00.167565 31667 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:00.180491 3033 consensus_queue.cc:1035] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:25:00.087460 3037 consensus_queue.cc:1035] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.084s
I20260805 22:25:00.186902 32594 pending_rounds.cc:76] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf: Aborting all ops after (but not including) 6
I20260805 22:25:00.187698 32594 pending_rounds.cc:97] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf: Aborting uncommitted NO_OP operation due to leader change: 6.7
I20260805 22:25:00.137938 2941 raft_consensus.cc:2781] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:00.187913 32594 raft_consensus.cc:2866] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:25:00.172356 2824 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:00.172262 2631 raft_consensus.cc:2726] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260805 22:25:00.193709 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 101665us
I20260805 22:25:00.193691 31667 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:00.221470 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:25:00.193265 2855 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.222425 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:25:00.223826 2855 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:00.191551 2867 consensus_queue.cc:1035] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260805 22:25:00.224220 31357 catalog_manager.cc:5495] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 3 to 5, leader changed from 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 5 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:25:00.195156 2824 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.189893 2941 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:00.230947 2633 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:00.232733 3010 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:00.237569 31444 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:00.241967 32587 raft_consensus.cc:2454] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:25:00.191864 2643 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:00.242830 31706 raft_consensus.cc:1226] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:25:00.242599 2953 raft_consensus.cc:3037] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Advancing to term 3
W20260805 22:25:00.252053 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 65916.5us
I20260805 22:25:00.235422 31357 catalog_manager.cc:5495] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:25:00.260108 31668 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:00.269150 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:00.269876 31601 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:25:00.274073 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:00.274859 31601 raft_consensus.cc:2379] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 8: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:00.277367 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:00.280891 31594 raft_consensus.cc:2379] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 8: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:00.283102 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:00.283659 31594 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
I20260805 22:25:00.284991 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:00.285792 31594 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
I20260805 22:25:00.285931 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:00.286772 31604 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:25:00.286981 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:00.288604 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:00.289083 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:00.289561 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:00.289747 31609 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:00.290130 31502 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:25:00.243348 2941 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:00.291088 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:00.291704 31594 raft_consensus.cc:3037] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:00.292582 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:00.293318 31596 raft_consensus.cc:2379] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 6: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:00.292433 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:00.295743 31603 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:00.295759 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:00.296394 31602 raft_consensus.cc:1261] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260805 22:25:00.296903 31600 raft_consensus.cc:2379] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 6: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:00.298590 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:00.300105 31608 raft_consensus.cc:1261] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260805 22:25:00.300762 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:00.273069 2643 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.309073 1989 raft_consensus.cc:3032] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Stepping down as leader of term 2
I20260805 22:25:00.310002 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:00.272046 2628 raft_consensus.cc:2781] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:25:00.312152 31486 raft_consensus.cc:1261] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260805 22:25:00.312084 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:00.289182 3044 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:00.319082 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:00.317899 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:00.316188 31495 raft_consensus.cc:1226] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:00.217159 2842 consensus_queue.cc:260] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.224241 2452 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.320571 31493 raft_consensus.cc:1226] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 6. Current term is 7. Ops: []
W20260805 22:25:00.319804 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 152456us
I20260805 22:25:00.331530 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:00.332118 31554 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:00.323974 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:00.333050 31486 raft_consensus.cc:3037] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:00.326020 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:00.221529 2840 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.334658 31594 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:25:00.329931 31596 raft_consensus.cc:1226] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 7. Current term is 8. Ops: []
I20260805 22:25:00.329842 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:00.336468 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:00.336848 31604 raft_consensus.cc:1226] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5. Ops: []
I20260805 22:25:00.267943 3010 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.350236 31602 raft_consensus.cc:2379] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 8: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:00.309520 1989 raft_consensus.cc:729] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:00.351078 31667 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:00.351783 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:00.396736 31715 raft_consensus.cc:2379] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 3: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:00.397622 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:00.360841 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:00.387524 31714 raft_consensus.cc:1261] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260805 22:25:00.353173 2955 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:00.398900 2955 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.399315 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:00.399698 2452 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:00.400348 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:00.400805 2955 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:00.390163 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:00.390331 31608 raft_consensus.cc:1261] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260805 22:25:00.392927 31708 raft_consensus.cc:1226] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 8. Current term is 9. Ops: []
I20260805 22:25:00.401435 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:00.365115 31609 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:25:00.330701 2628 raft_consensus.cc:686] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:00.393671 31709 raft_consensus.cc:1226] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5. Ops: []
I20260805 22:25:00.394435 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 3 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:00.331007 2956 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:00.414312 2956 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.405158 2867 consensus_queue.cc:1035] T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20260805 22:25:00.406466 31357 catalog_manager.cc:5495] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:25:00.416374 31357 catalog_manager.cc:5495] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 2 to 3, leader changed from 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 3 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
W20260805 22:25:00.396605 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 58943.4us
I20260805 22:25:00.328862 31601 raft_consensus.cc:1261] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
I20260805 22:25:00.418713 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:00.418954 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:00.419667 2628 consensus_queue.cc:237] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.420428 31711 raft_consensus.cc:1261] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 4. (index mismatch)
I20260805 22:25:00.336755 31492 raft_consensus.cc:1226] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 6. Current term is 7. Ops: []
I20260805 22:25:00.421723 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:00.387678 2948 raft_consensus.cc:2781] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:00.398597 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:00.418041 31357 catalog_manager.cc:5495] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:25:00.403596 31444 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:00.426179 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:00.426795 31714 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
I20260805 22:25:00.404829 31600 raft_consensus.cc:1226] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 6. Current term is 7. Ops: []
I20260805 22:25:00.404666 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:00.404733 31494 raft_consensus.cc:2379] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 8: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:00.407318 32529 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:00.408833 3051 log.cc:924] Time spent T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260805 22:25:00.431778 2824 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:00.432732 2956 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:00.409569 1989 consensus_queue.cc:260] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.433724 2840 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:00.434067 2993 raft_consensus.cc:2781] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader election won for term 8
I20260805 22:25:00.434553 2993 raft_consensus.cc:686] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 8 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:00.435364 2993 consensus_queue.cc:237] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.406857 2716 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:00.433769 1989 raft_consensus.cc:3037] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:25:00.438016 2948 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:00.438429 2948 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:00.412556 31603 raft_consensus.cc:2454] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:25:00.413663 2643 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:00.415449 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:00.405342 2855 raft_consensus.cc:2781] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:00.442314 31704 raft_consensus.cc:2379] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 9: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:00.444456 2953 raft_consensus.cc:2726] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:25:00.445304 31713 raft_consensus.cc:1226] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:25:00.442713 31357 catalog_manager.cc:5495] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 6 to 7, leader changed from 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 7 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:25:00.447412 31357 catalog_manager.cc:5495] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:25:00.423262 31667 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:00.442279 2855 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:00.346762 31594 raft_consensus.cc:1226] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 6. Current term is 7. Ops: []
W20260805 22:25:00.448479 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 51527.5us
I20260805 22:25:00.397763 31602 raft_consensus.cc:1226] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:00.450096 31357 catalog_manager.cc:5495] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 2 to 5, leader changed from 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 5 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:25:00.414939 2884 consensus_queue.cc:1035] T 52af4d94f396434d997b46f52763cd20 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.038s
I20260805 22:25:00.452736 31493 raft_consensus.cc:1261] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260805 22:25:00.443343 31705 raft_consensus.cc:1261] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260805 22:25:00.451507 31357 catalog_manager.cc:5495] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 6 to 7, leader changed from 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 7 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:25:00.398594 3010 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:00.466082 31554 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:00.437882 2716 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.458635 3026 consensus_queue.cc:1035] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:25:00.462003 2631 consensus_queue.cc:1046] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:25:00.463006 2956 consensus_queue.cc:1046] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:25:00.467880 31490 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:25:00.504361 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:00.505193 32594 raft_consensus.cc:2379] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 8: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:00.465502 3042 raft_consensus.cc:2726] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:25:00.516328 31359 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34394 (request call id 296) took 10846 ms. Trace:
I20260805 22:25:00.508150 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:00.457237 2953 raft_consensus.cc:2726] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:25:00.510013 32587 raft_consensus.cc:1261] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260805 22:25:00.527424 31499 raft_consensus.cc:1261] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:25:00.510013 31704 raft_consensus.cc:1261] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260805 22:25:00.528345 31705 raft_consensus.cc:1226] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 7. Current term is 8. Ops: []
I20260805 22:25:00.517740 31490 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:25:00.453533 2855 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:00.475708 3032 raft_consensus.cc:2781] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:00.465206 2941 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:00.533572 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 67926.4us
I20260805 22:25:00.532778 3055 consensus_queue.cc:1046] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 3, Time since last communication: 0.032s
I20260805 22:25:00.549974 31667 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:00.516723 31359 rpcz_store.cc:276] 0805 22:24:49.669821 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:49.735180 (+ 65359us) service_pool.cc:227] Handling call
0805 22:25:00.516249 (+10781069us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0805 22:24:59.178762 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0805 22:24:59.201794 (+ 23032us) write_op.cc:432] Acquiring schema lock in shared mode
0805 22:24:59.201847 (+ 53us) write_op.cc:435] Acquired schema lock
0805 22:24:59.201863 (+ 16us) tablet.cc:660] Decoding operations
0805 22:24:59.202803 (+ 940us) write_op.cc:620] Acquiring the partition lock for write op
0805 22:24:59.202888 (+ 85us) write_op.cc:641] Partition lock acquired for write op
0805 22:24:59.202915 (+ 27us) tablet.cc:683] Acquiring locks for 7 operations
0805 22:24:59.203589 (+ 674us) tablet.cc:699] Row locks acquired
0805 22:24:59.203625 (+ 36us) write_op.cc:260] PREPARE: finished
0805 22:24:59.203791 (+ 166us) write_op.cc:270] Start()
0805 22:24:59.203934 (+ 143us) write_op.cc:276] Timestamp: P: 1785968699203725 usec, L: 0
0805 22:24:59.203950 (+ 16us) op_driver.cc:331] REPLICATION: starting
0805 22:24:59.204620 (+ 670us) log.cc:842] Serialized 2865 byte log entry
0805 22:24:59.568573 (+363953us) op_driver.cc:444] REPLICATION: finished
0805 22:24:59.647106 (+ 78533us) write_op.cc:301] APPLY: starting
0805 22:24:59.647184 (+ 78us) tablet.cc:1365] starting BulkCheckPresence
0805 22:24:59.647350 (+ 166us) tablet.cc:1368] finished BulkCheckPresence
0805 22:24:59.647462 (+ 112us) tablet.cc:1370] starting ApplyRowOperation cycle
0805 22:24:59.710089 (+ 62627us) tablet.cc:1381] finished ApplyRowOperation cycle
0805 22:24:59.710304 (+ 215us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=7
0805 22:24:59.710345 (+ 41us) write_op.cc:312] APPLY: finished
0805 22:24:59.710578 (+ 233us) log.cc:842] Serialized 59 byte log entry
0805 22:24:59.710741 (+ 163us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:59.710848 (+ 107us) write_op.cc:454] Released schema lock
0805 22:24:59.710889 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0805 22:24:59.711186 (+ 297us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:24:59.711226 (+ 40us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":212951,"reported_tablets":34,"child_traces":[["op",{"apply.queue_time_us":78242,"apply.run_cpu_time_us":3078,"apply.run_wall_time_us":64436,"num_ops":7,"prepare.queue_time_us":158680,"prepare.run_cpu_time_us":15543,"prepare.run_wall_time_us":222235,"raft.queue_time_us":157819,"raft.run_cpu_time_us":16348,"raft.run_wall_time_us":74772,"replication_time_us":364521,"thread_start_us":451974,"threads_started":4,"wal-append.queue_time_us":189472}]]}
I20260805 22:25:00.547241 3010 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:00.558203 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:00.579115 31705 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:25:00.548538 3032 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:00.555997 2891 raft_consensus.cc:3037] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:25:00.587728 2628 raft_consensus.cc:2781] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election won for term 8
I20260805 22:25:00.509843 2945 consensus_queue.cc:1035] T ab66683a9e5e46c2b09be4afdee2a08c P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.048s
I20260805 22:25:00.599192 3044 raft_consensus.cc:2726] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:00.603111 2941 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:00.608505 31708 raft_consensus.cc:1261] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:25:00.633369 31667 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:00.635054 31667 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:00.587378 3032 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:00.547672 3042 consensus_queue.cc:1035] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.103s
I20260805 22:25:00.633977 2628 raft_consensus.cc:686] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:00.645988 3055 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:00.585747 3010 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.652617 2953 raft_consensus.cc:2726] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260805 22:25:00.703759 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 198192us
W20260805 22:25:00.707931 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 173483us
I20260805 22:25:00.709395 31444 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:00.657800 2452 raft_consensus.cc:2726] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260805 22:25:00.712690 31444 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:00.678808 2891 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:00.658613 2956 consensus_queue.cc:1046] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260805 22:25:00.747510 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:00.715538 2864 raft_consensus.cc:2781] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:25:00.727105 2941 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:00.742360 2731 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:00.757699 2731 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.759874 2731 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:00.762600 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 48527.7us
I20260805 22:25:00.704620 3055 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.768553 31444 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
W20260805 22:25:00.761853 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 126888us
I20260805 22:25:00.773211 31667 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:00.775918 31486 raft_consensus.cc:2454] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:25:00.778465 2948 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.779995 2948 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:00.750255 2716 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:00.783421 2716 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:00.783929 2716 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.785279 2631 raft_consensus.cc:2726] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:00.785523 2452 consensus_queue.cc:1035] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260805 22:25:00.742334 2891 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.788255 2891 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:00.788853 3025 consensus_queue.cc:1046] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.140s
I20260805 22:25:00.755427 2956 consensus_queue.cc:1225] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer 516cb07ce3064ccbbca35e912d62816c log is divergent from this leader: its last log entry 5.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 5
I20260805 22:25:00.755021 2864 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
W20260805 22:25:00.793494 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 79002us
I20260805 22:25:00.794153 31554 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:00.761703 2941 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:00.746366 2855 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.796475 31444 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:00.752236 2945 consensus_queue.cc:1035] T 3095cdb25afc4c26bd9d1e03a919b474 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260805 22:25:00.803718 31444 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:00.797279 2941 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:00.795719 2864 raft_consensus.cc:3037] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:25:00.808446 3025 raft_consensus.cc:3032] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 LEADER]: Stepping down as leader of term 7
I20260805 22:25:00.808866 3025 raft_consensus.cc:729] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:00.797133 2631 consensus_queue.cc:1035] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.012s
I20260805 22:25:00.808390 3078 raft_consensus.cc:2781] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:25:00.809742 3078 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:00.810093 3078 raft_consensus.cc:3037] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:00.812620 2855 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:00.813596 31423 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica f8e9f4c756da4203917c464e2b179994 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 4 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:00.813627 2716 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:00.816143 3067 log.cc:924] Time spent T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.066s user 0.000s sys 0.000s
I20260805 22:25:00.816041 2941 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.816422 3025 consensus_queue.cc:260] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.817704 3025 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:00.781409 2867 consensus_queue.cc:1035] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.024s
I20260805 22:25:00.795207 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:00.825229 3044 raft_consensus.cc:2781] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:25:00.825008 2716 consensus_queue.cc:1035] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:25:00.825748 3044 raft_consensus.cc:686] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:00.825763 31444 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:00.826597 3044 consensus_queue.cc:237] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.797976 2628 consensus_queue.cc:237] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.791349 2731 consensus_queue.cc:1046] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 7.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20260805 22:25:00.830578 2731 raft_consensus.cc:3032] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 7 LEADER]: Stepping down as leader of term 7
I20260805 22:25:00.830932 2731 raft_consensus.cc:729] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:00.832074 2643 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:25:00.831758 2731 consensus_queue.cc:260] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.833122 31444 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:00.754033 3068 consensus_queue.cc:1035] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.056s
I20260805 22:25:00.844508 31444 leader_election.cc:398] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5.
I20260805 22:25:00.807741 3042 raft_consensus.cc:2781] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:25:00.802294 3074 raft_consensus.cc:2726] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:25:00.846585 2941 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:00.845446 31444 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
W20260805 22:25:00.848794 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 56692.2us
I20260805 22:25:00.843927 2643 raft_consensus.cc:2781] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:25:00.861598 2643 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:00.862052 2643 raft_consensus.cc:3037] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:00.865471 2971 raft_consensus.cc:2781] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:00.865896 2971 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:00.866192 2971 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:00.868055 3032 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.869582 2452 consensus_queue.cc:1046] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:25:00.870790 3077 raft_consensus.cc:3032] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:25:00.871218 3077 raft_consensus.cc:729] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:00.871827 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:00.871987 3077 consensus_queue.cc:260] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.873291 3077 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:00.873172 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:00.870620 3032 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:00.877492 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:00.878373 31602 raft_consensus.cc:2396] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 8 because replica has last-logged OpId of term: 7 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 4.
I20260805 22:25:00.879827 31444 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:00.805589 3045 consensus_queue.cc:1035] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.046s
I20260805 22:25:00.849401 2956 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:25:00.885207 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 67013.5us
I20260805 22:25:00.878288 3026 raft_consensus.cc:3032] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:25:00.887027 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:00.887648 31603 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:00.888450 31667 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:00.881318 3042 raft_consensus.cc:686] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
W20260805 22:25:00.894068 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 5df9cd03cde746e0956c8c7726d4eb70, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:00.895841 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 5df9cd03cde746e0956c8c7726d4eb70, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.904772 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:00.900787 31444 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
W20260805 22:25:00.905815 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 103807us
I20260805 22:25:00.880348 3010 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:00.907343 31501 raft_consensus.cc:1226] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 7. Current term is 8. Ops: []
W20260805 22:25:00.914487 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 5e4ad7072aab4331878e988a26a6d887, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.886076 2864 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.908541 31603 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:00.892431 3026 raft_consensus.cc:729] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:00.885296 2956 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.892325 2941 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:00.854182 2953 consensus_queue.cc:1046] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.038s
I20260805 22:25:00.898453 2631 raft_consensus.cc:2781] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:25:00.913340 2452 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
W20260805 22:25:00.914803 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 5e4ad7072aab4331878e988a26a6d887, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.924875 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:00.925697 31444 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:00.927304 31493 raft_consensus.cc:1226] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
I20260805 22:25:00.914359 3068 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:00.931696 3068 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.934028 3068 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:00.934564 2956 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:00.935379 3055 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:00.936151 3087 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:00.936280 3078 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.936856 3080 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:00.937497 3080 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.939653 3080 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:00.940174 3080 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:00.940614 3080 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.936769 3087 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.898200 2855 consensus_queue.cc:1046] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.044s
I20260805 22:25:00.942817 3077 raft_consensus.cc:3032] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:25:00.943261 3077 raft_consensus.cc:729] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:00.943333 2631 raft_consensus.cc:686] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:00.944175 3077 consensus_queue.cc:260] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.944314 2631 consensus_queue.cc:237] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.943359 3087 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:00.922773 3074 raft_consensus.cc:2741] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
W20260805 22:25:00.948103 3088 log.cc:924] Time spent T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.101s user 0.001s sys 0.000s
I20260805 22:25:00.948961 3026 consensus_queue.cc:260] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:00.945550 3077 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:00.945132 3042 consensus_queue.cc:237] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.948954 3080 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:00.925071 2941 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:00.954821 2941 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
W20260805 22:25:00.967815 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 4deaa72a885a43bfbc6171a7ed13ad81, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:00.968179 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 4deaa72a885a43bfbc6171a7ed13ad81, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:00.968837 31444 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:00.971760 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:00.972165 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:00.972249 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:00.973004 31500 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:00.973839 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:00.974481 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:00.975829 31502 raft_consensus.cc:1226] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 7. Current term is 8. Ops: []
I20260805 22:25:00.977290 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:00.981050 31500 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:25:00.986402 31667 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:00.993258 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:00.992264 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:00.950945 2452 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.000495 31603 raft_consensus.cc:1226] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 6. Current term is 7. Ops: []
I20260805 22:25:01.005066 31667 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:00.965942 3045 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:00.968140 3026 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:01.030577 31667 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:01.030906 31492 raft_consensus.cc:1261] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 4. (term mismatch)
I20260805 22:25:01.037906 31492 pending_rounds.cc:76] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994: Aborting all ops after (but not including) 3
I20260805 22:25:01.038178 31492 pending_rounds.cc:97] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994: Aborting uncommitted NO_OP operation due to leader change: 7.4
I20260805 22:25:01.038426 31492 raft_consensus.cc:2866] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:25:01.056751 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:01.065989 31706 raft_consensus.cc:3037] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
W20260805 22:25:01.068364 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 99574.7us
I20260805 22:25:01.069325 31444 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:00.993958 2855 raft_consensus.cc:2781] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election won for term 7
W20260805 22:25:01.099184 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 149354us
I20260805 22:25:00.966212 2867 consensus_queue.cc:1035] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.067s
I20260805 22:25:01.113106 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:01.009598 3044 raft_consensus.cc:2726] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:25:01.128556 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:01.029723 2956 raft_consensus.cc:2726] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:01.149844 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:01.154600 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:01.096768 3042 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
W20260805 22:25:01.164350 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 527d60ab6db74595bed7e87ca2ecc6e8, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:01.164724 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 527d60ab6db74595bed7e87ca2ecc6e8, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.109797 3096 raft_consensus.cc:2726] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:01.139772 2762 raft_consensus.cc:3032] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:25:01.165894 31554 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:01.146648 3033 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:01.157435 3044 raft_consensus.cc:2726] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260805 22:25:01.191215 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:01.027954 3074 consensus_queue.cc:1035] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.161s
I20260805 22:25:01.196327 2985 raft_consensus.cc:2726] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:25:01.202236 2643 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.164055 2956 consensus_queue.cc:1046] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.042s
I20260805 22:25:01.206111 3072 consensus_queue.cc:1035] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.246s
W20260805 22:25:01.207585 3094 log.cc:924] Time spent T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.207s user 0.001s sys 0.000s
I20260805 22:25:01.211323 31704 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:25:01.183264 2762 raft_consensus.cc:729] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:01.226614 2956 raft_consensus.cc:3032] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 LEADER]: Stepping down as leader of term 8
I20260805 22:25:01.227038 2956 raft_consensus.cc:729] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:01.227851 2956 consensus_queue.cc:260] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.229099 2956 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:01.230989 3118 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:01.231587 3118 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:01.242653 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 150045us
I20260805 22:25:01.248109 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:01.254698 31500 raft_consensus.cc:2379] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 4: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:01.218350 3078 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:01.259295 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:01.260290 31602 raft_consensus.cc:1226] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:01.187813 2971 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.190032 3042 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.244035 2643 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:01.268353 2955 raft_consensus.cc:3032] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:25:01.274466 2955 raft_consensus.cc:729] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:01.275569 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:01.276320 31602 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:25:01.276505 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:01.278393 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:01.279240 31667 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:01.279718 31502 raft_consensus.cc:2354] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 6. Current term is 7.
I20260805 22:25:01.213876 3033 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.281811 3121 raft_consensus.cc:2781] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:01.208721 3096 consensus_queue.cc:1046] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.071s
I20260805 22:25:01.292778 31706 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:25:01.299105 3033 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:01.291875 3121 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:01.301761 3117 consensus_queue.cc:1046] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.023s
I20260805 22:25:01.302970 2762 consensus_queue.cc:260] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.309036 31500 raft_consensus.cc:1226] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:01.310961 31713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58348 (request call id 577) took 1077 ms. Trace:
I20260805 22:25:01.311154 2628 consensus_queue.cc:1046] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
W20260805 22:25:01.315155 3091 log.cc:924] Time spent T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.467s user 0.005s sys 0.000s
I20260805 22:25:01.311388 3121 raft_consensus.cc:3037] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:01.317968 2731 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:01.311422 31713 rpcz_store.cc:276] 0805 22:25:00.233989 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:00.234213 (+ 224us) service_pool.cc:227] Handling call
0805 22:25:00.266177 (+ 31964us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:00.453206 (+187029us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:01.310714 (+857508us) spinlock_profiling.cc:243] Waited 179 ms on lock 0x7b5400032fd4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:01.310895 (+ 181us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":411557632}
I20260805 22:25:01.323179 31556 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:01.273098 3118 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:01.360864 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:01.361869 31706 raft_consensus.cc:2454] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:25:01.360873 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:01.378484 2955 consensus_queue.cc:260] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.380159 2955 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:01.393214 3077 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:01.393957 3077 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.395143 2864 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:01.395824 2716 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:01.396337 2716 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.399466 2716 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:01.353029 3037 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:25:01.301117 2941 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.400058 2762 raft_consensus.cc:3037] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:01.410197 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:01.411757 31493 raft_consensus.cc:1261] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:25:01.406035 3077 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:01.414129 3124 log.cc:924] Time spent T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.054s user 0.000s sys 0.000s
I20260805 22:25:01.414762 3036 raft_consensus.cc:3032] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 7 LEADER]: Stepping down as leader of term 7
I20260805 22:25:01.415174 3036 raft_consensus.cc:729] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:01.416006 3036 consensus_queue.cc:260] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:01.417658 3099 log.cc:924] Time spent T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.180s user 0.001s sys 0.000s
I20260805 22:25:01.420864 3077 consensus_queue.cc:1035] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:25:01.434351 3042 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:01.435966 1989 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:01.436632 1989 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.439386 1989 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:01.358880 3007 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:01.441895 3007 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.444641 3007 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:01.357781 2452 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:01.448813 3045 raft_consensus.cc:2726] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:25:01.365893 3008 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:01.453781 3008 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.455893 3008 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:01.456941 2971 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:01.459757 31704 raft_consensus.cc:1261] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 5 index: 6. (term mismatch)
I20260805 22:25:01.460156 31704 pending_rounds.cc:76] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385: Aborting all ops after (but not including) 5
I20260805 22:25:01.460346 31704 pending_rounds.cc:97] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385: Aborting uncommitted NO_OP operation due to leader change: 4.6
I20260805 22:25:01.460537 31704 raft_consensus.cc:2866] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:25:01.461581 31594 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
W20260805 22:25:01.462628 3066 log.cc:924] Time spent T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.870s user 0.005s sys 0.008s
I20260805 22:25:01.464047 31706 raft_consensus.cc:1226] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 3. Current term is 4. Ops: []
W20260805 22:25:01.465760 3062 log.cc:924] Time spent T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.722s user 0.007s sys 0.006s
I20260805 22:25:01.466404 3025 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260805 22:25:01.467726 31707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 2298) took 1132 ms. Trace:
I20260805 22:25:01.468099 31707 rpcz_store.cc:276] 0805 22:25:00.335761 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:00.421016 (+ 85255us) service_pool.cc:227] Handling call
0805 22:25:00.421349 (+ 333us) raft_consensus.cc:1457] Updating replica for 2 ops
0805 22:25:00.431924 (+ 10575us) raft_consensus.cc:1520] Early marking committed up to index 4
0805 22:25:00.432260 (+ 336us) log.cc:842] Serialized 14 byte log entry
0805 22:25:00.547624 (+115364us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0805 22:25:00.548191 (+ 567us) log.cc:842] Serialized 52 byte log entry
0805 22:25:00.550041 (+ 1850us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:25:00.556158 (+ 6117us) log.cc:842] Serialized 14 byte log entry
0805 22:25:00.582947 (+ 26789us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:00.950035 (+367088us) spinlock_profiling.cc:243] Waited 10.1 ms on lock 0x7b54002b3ed4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:00.950081 (+ 46us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:01.465050 (+514969us) spinlock_profiling.cc:243] Waited 5.95 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:25:01.467496 (+ 2446us) raft_consensus.cc:1671] finished
0805 22:25:01.467517 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:01.467668 (+ 151us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":55566,"prepare.queue_time_us":1891,"prepare.run_cpu_time_us":73,"prepare.run_wall_time_us":79,"raft.queue_time_us":28045,"raft.run_cpu_time_us":223,"raft.run_wall_time_us":9419,"spinlock_wait_cycles":37040128,"thread_start_us":115070,"threads_started":1,"wal-append.queue_time_us":226129}
I20260805 22:25:01.417212 3036 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:01.472916 31494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40740 (request call id 1279) took 1184 ms. Trace:
I20260805 22:25:01.473309 31494 rpcz_store.cc:276] 0805 22:25:00.288208 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:00.288362 (+ 154us) service_pool.cc:227] Handling call
0805 22:25:01.472651 (+1184289us) spinlock_profiling.cc:243] Waited 107 ms on lock 0x7b54000c17d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:01.472816 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":245912320}
I20260805 22:25:01.475970 31500 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260805 22:25:01.479838 31499 raft_consensus.cc:1226] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 7. Current term is 8. Ops: []
W20260805 22:25:01.481235 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 315415us
I20260805 22:25:01.492503 32579 raft_consensus.cc:1226] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6. Ops: []
W20260805 22:25:01.482546 3032 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.518s user 0.014s sys 0.026s
I20260805 22:25:01.498420 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:01.492913 31554 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:01.499615 2941 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:01.509626 3025 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.001s
I20260805 22:25:01.512413 3025 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:01.505497 3037 raft_consensus.cc:2781] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:25:01.514962 3037 raft_consensus.cc:686] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [term 7 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:01.515970 3037 consensus_queue.cc:237] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.517624 3047 consensus_queue.cc:1035] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:25:01.520547 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
W20260805 22:25:01.516587 3115 log.cc:924] Time spent T c5b0fd7f6d7a46b2a77e841ceda358ef P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.269s user 0.000s sys 0.001s
I20260805 22:25:01.524309 32591 raft_consensus.cc:1226] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5. Ops: []
I20260805 22:25:01.527978 2842 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:01.513126 3025 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.532589 3025 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:01.532931 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:01.538970 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:01.539680 31502 raft_consensus.cc:2379] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 8: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:01.539827 31554 leader_election.cc:398] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Vote denied by peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607) with higher term. Message: Invalid argument: T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 6. Current term is 7.
I20260805 22:25:01.540102 31554 leader_election.cc:401] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260805 22:25:01.533771 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:01.535442 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:01.542694 32584 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:25:01.543363 31667 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:01.536445 32591 raft_consensus.cc:1261] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:25:01.546775 3121 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.549106 3121 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:01.551028 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:01.587061 32591 raft_consensus.cc:2379] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 6: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:01.552026 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:01.552142 32584 raft_consensus.cc:1261] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 8 index: 8. (index mismatch)
I20260805 22:25:01.552799 3068 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:01.552747 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:01.553550 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:01.551069 2731 consensus_queue.cc:1046] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260805 22:25:01.553300 3036 raft_consensus.cc:2726] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607) with higher term. Message: Invalid argument: T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 6. Current term is 7.
I20260805 22:25:01.554325 2956 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:01.555874 2891 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:01.556234 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:01.559401 3026 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:01.562052 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:01.564883 31668 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:01.584717 2716 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:01.574048 31490 raft_consensus.cc:1226] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 6. Current term is 7. Ops: []
W20260805 22:25:01.495688 3032 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.535s user 0.014s sys 0.026s
I20260805 22:25:01.487238 2855 raft_consensus.cc:686] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:01.595964 32589 raft_consensus.cc:2379] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 6: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:01.595830 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:01.599735 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:01.600135 31554 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:01.600769 31495 raft_consensus.cc:1226] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5. Ops: []
I20260805 22:25:01.617599 31554 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:01.572567 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:01.607699 31711 raft_consensus.cc:3037] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:25:01.615361 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 50598.1us
I20260805 22:25:01.604821 31502 raft_consensus.cc:3037] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:01.620493 31668 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:01.637825 3008 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:01.638427 3008 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.639418 31554 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:01.640727 3008 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:01.647814 32529 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:25:01.651324 3008 raft_consensus.cc:2781] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:01.652019 3008 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:01.652455 3008 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:01.655562 31668 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:01.657867 3127 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:01.659272 3032 raft_consensus.cc:2726] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:25:01.659267 2762 raft_consensus.cc:2726] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:25:01.615814 3026 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.660319 2855 consensus_queue.cc:237] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.662096 3026 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:01.663520 31707 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 9.
I20260805 22:25:01.664013 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:01.664726 31600 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 9.
I20260805 22:25:01.664193 3052 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:01.666193 31667 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:01.612327 2891 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.627050 3121 raft_consensus.cc:2726] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader election lost for term 3. Reason: could not achieve majority
I20260805 22:25:01.632609 31596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1272) took 1266 ms. Trace:
I20260805 22:25:01.624167 2731 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:01.624856 3036 raft_consensus.cc:2781] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:01.675419 31668 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:25:01.624701 2953 raft_consensus.cc:2726] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:25:01.632674 3047 raft_consensus.cc:2726] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:01.680465 2762 consensus_queue.cc:1035] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:25:01.680986 2731 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.674289 3127 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.603547 3068 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.684873 31502 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:25:01.690915 31668 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:01.619392 2716 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.675954 31596 rpcz_store.cc:276] 0805 22:25:00.351745 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:00.356684 (+ 4939us) service_pool.cc:227] Handling call
0805 22:25:00.395382 (+ 38698us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:00.395578 (+ 196us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:00.395625 (+ 47us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:00.422375 (+ 26750us) log.cc:842] Serialized 26 byte log entry
0805 22:25:00.529756 (+107381us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:25:00.539447 (+ 9691us) log.cc:842] Serialized 14 byte log entry
0805 22:25:01.278706 (+739259us) spinlock_profiling.cc:243] Waited 3.02 ms on lock 0x7b1800204700. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e10df96 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e122345 00007f2736f59a39
0805 22:25:01.308954 (+ 30248us) spinlock_profiling.cc:243] Waited 7.89 ms on lock 0x7b1800204700. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e122345 00007f2736f59a39 00007f272e13f8bb 00007f272e13dcbf 00007f272e13d155 00007f27310603d0 00007f2731072576 00007f27310830d7 00007f2731083058
0805 22:25:01.387377 (+ 78423us) spinlock_profiling.cc:243] Waited 21.7 ms on lock 0x7b1800204700. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e10df96 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e140638 00007f272e143b11
0805 22:25:01.523969 (+136592us) spinlock_profiling.cc:243] Waited 1.75 ms on lock 0x7b1800204700. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e140638 00007f272e143b11 00007f272e143ac9 00007f272e143a59 00007f272e143a21 00007f272e142d1d 00007f27373a4204 00007f27373a4039 00007f272e122dc6
0805 22:25:01.618134 (+ 94165us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:01.618227 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:01.618269 (+ 42us) raft_consensus.cc:1671] finished
0805 22:25:01.618280 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:01.618392 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7857,"prepare.queue_time_us":9286,"prepare.run_cpu_time_us":48,"prepare.run_wall_time_us":5314,"raft.queue_time_us":37578,"raft.run_cpu_time_us":114,"raft.run_wall_time_us":3533,"spinlock_wait_cycles":79001728,"thread_start_us":412929,"threads_started":2,"wal-append.queue_time_us":154871}
I20260805 22:25:01.677699 3036 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:01.697926 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:01.677168 2631 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260805 22:25:01.701247 31711 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:25:01.702440 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:25:01.708376 32529 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:01.684263 3026 raft_consensus.cc:2781] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:25:01.723187 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:01.692018 3125 consensus_queue.cc:1035] T 91a9acdbc98a4e799374fe6e8950c3a7 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.032s
I20260805 22:25:01.674656 3052 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.725514 3007 consensus_queue.cc:1046] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:25:01.726837 3052 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:01.728345 3139 consensus_queue.cc:1046] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.033s
I20260805 22:25:01.705885 3036 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:01.735992 2731 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:01.740437 3127 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:01.709854 3080 consensus_queue.cc:1035] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.032s
I20260805 22:25:01.715737 3026 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:01.743048 3026 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:01.706024 2631 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:01.716741 2955 raft_consensus.cc:2726] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:01.756089 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:01.756826 32589 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:25:01.757109 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:01.759323 31554 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:01.760996 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:01.761313 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:01.768556 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:01.758632 3137 consensus_queue.cc:1046] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.026s
I20260805 22:25:01.779495 3010 raft_consensus.cc:3032] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:25:01.779917 3010 raft_consensus.cc:729] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:01.780737 3010 consensus_queue.cc:260] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.782187 3010 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:01.783100 2985 raft_consensus.cc:2781] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:25:01.783609 31602 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:25:01.783728 2985 raft_consensus.cc:686] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:01.754751 2631 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Advancing to term 10
W20260805 22:25:01.791599 31444 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.671s user 0.019s sys 0.065s
W20260805 22:25:01.791956 31444 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.672s user 0.019s sys 0.065s
W20260805 22:25:01.792537 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 723250us
I20260805 22:25:01.784883 3078 raft_consensus.cc:3032] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:25:01.793068 3078 raft_consensus.cc:729] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:01.794041 32581 raft_consensus.cc:1261] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260805 22:25:01.793917 3078 consensus_queue.cc:260] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.794692 2956 raft_consensus.cc:2726] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:01.757071 3080 consensus_queue.cc:1046] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260805 22:25:01.795262 3078 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:01.795647 3080 raft_consensus.cc:3032] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:25:01.795990 3080 raft_consensus.cc:729] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:01.797447 31554 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:25:01.798346 2731 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260805 22:25:01.798715 2731 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:01.799016 2731 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:01.805474 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:01.806187 31495 raft_consensus.cc:2454] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:25:01.806936 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:01.807952 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:01.808631 2876 raft_consensus.cc:3032] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 7 LEADER]: Stepping down as leader of term 7
I20260805 22:25:01.809046 2876 raft_consensus.cc:729] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:01.810032 31667 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:01.809911 2876 consensus_queue.cc:260] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.810906 31704 raft_consensus.cc:1226] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 7. Current term is 8. Ops: []
I20260805 22:25:01.811656 3025 raft_consensus.cc:2781] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:25:01.812137 3025 raft_consensus.cc:686] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:01.812959 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:01.814116 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:01.812347 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:01.814668 31492 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:25:01.814797 31502 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:25:01.815968 2716 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:01.817189 3068 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:01.808548 31495 raft_consensus.cc:2454] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:25:01.793284 31444 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:01.818742 31556 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:01.820434 31556 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:01.820807 31594 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260805 22:25:01.821916 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:01.824637 31556 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:01.827139 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:01.837030 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:01.839937 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:01.841959 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:01.842726 31713 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:25:01.839638 31554 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:01.844465 31556 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:01.845418 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:01.818240 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:01.846522 32529 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:01.848119 2975 raft_consensus.cc:2781] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:01.848487 2975 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:01.848809 2975 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:01.811093 2876 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:01.846724 3128 raft_consensus.cc:2726] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:01.840067 2948 raft_consensus.cc:2726] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:01.807860 3008 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.851887 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:01.852536 32581 raft_consensus.cc:2454] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:25:01.852905 3008 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:01.796800 3080 consensus_queue.cc:260] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.849948 31444 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:01.856747 31554 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:01.855952 31715 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:25:01.862879 31444 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:01.864631 31444 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:01.866840 2842 raft_consensus.cc:2726] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:01.885912 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:01.886246 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:01.886662 31708 raft_consensus.cc:2454] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
I20260805 22:25:01.886806 31709 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:25:01.868510 31715 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260805 22:25:01.869611 31554 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:01.839615 1989 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:01.846473 3032 raft_consensus.cc:2726] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:01.885097 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:01.918483 31711 raft_consensus.cc:2454] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:25:01.867156 31444 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:01.842031 2945 raft_consensus.cc:2726] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:25:01.888327 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:01.853412 3068 raft_consensus.cc:2726] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:01.891182 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:01.784830 2985 consensus_queue.cc:237] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:01.921743 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 54618.2us
I20260805 22:25:01.928514 31444 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:01.929944 3126 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:01.876428 3037 raft_consensus.cc:2781] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:01.917482 3007 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:01.855396 3087 consensus_queue.cc:1035] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.070s
I20260805 22:25:01.860818 2891 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:01.930702 32594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57382 (request call id 740) took 1177 ms. Trace:
I20260805 22:25:01.930891 3142 raft_consensus.cc:2781] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:25:01.926891 3032 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:01.933326 3032 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.935259 3042 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260805 22:25:01.936220 3042 consensus_queue.cc:1035] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:25:01.920758 1989 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.937784 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:01.939510 3133 log.cc:924] Time spent T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.051s user 0.000s sys 0.001s
I20260805 22:25:01.940234 32581 raft_consensus.cc:1226] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 6. Current term is 7. Ops: []
I20260805 22:25:01.846840 3025 consensus_queue.cc:237] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.917344 3137 raft_consensus.cc:2781] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:25:01.869858 3128 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:01.878405 3008 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:01.868363 3045 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:25:01.944235 1989 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:01.945138 2762 consensus_queue.cc:1046] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.013s
I20260805 22:25:01.971885 2762 raft_consensus.cc:3032] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:25:01.946375 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:01.948628 3137 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:01.949406 3128 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:01.959466 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:01.949973 3008 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.951275 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:01.978562 3137 raft_consensus.cc:3037] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:01.980957 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:25:01.925854 3088 log.cc:924] Time spent T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20260805 22:25:01.951565 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:01.961874 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:01.951977 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:01.951346 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:01.953305 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:01.931247 3007 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.931927 32594 rpcz_store.cc:276] 0805 22:25:00.753229 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:00.753359 (+ 130us) service_pool.cc:227] Handling call
0805 22:25:00.753719 (+ 360us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:01.633505 (+879786us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:01.633625 (+ 120us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:01.658089 (+ 24464us) log.cc:842] Serialized 26 byte log entry
0805 22:25:01.827897 (+169808us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:25:01.827933 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:01.828000 (+ 67us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:01.930502 (+102502us) raft_consensus.cc:1671] finished
0805 22:25:01.930521 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:01.930649 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":633661,"raft.queue_time_us":602733,"raft.run_cpu_time_us":142,"raft.run_wall_time_us":4871,"thread_start_us":484085,"threads_started":2,"wal-append.queue_time_us":270023}
I20260805 22:25:01.963797 3036 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:01.954196 2631 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.956718 2731 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:01.969628 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:02.004622 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:01.970165 3032 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:01.885943 3119 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:01.881664 3131 raft_consensus.cc:2726] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:25:01.952292 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:02.028908 2731 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:02.033175 3119 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:02.034360 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:02.035110 31602 raft_consensus.cc:2379] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 7: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:02.034819 3137 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:02.035938 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:01.983493 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:02.038367 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:01.982059 31715 raft_consensus.cc:2396] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 5.
I20260805 22:25:01.986949 3080 raft_consensus.cc:3037] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:01.999567 31711 raft_consensus.cc:1261] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260805 22:25:01.927260 3079 raft_consensus.cc:2726] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260805 22:25:01.930658 3037 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:02.042995 3037 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:02.044428 31597 raft_consensus.cc:1226] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 9. Current term is 10. Ops: []
I20260805 22:25:01.925593 3125 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:01.972328 2762 raft_consensus.cc:729] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:01.930632 3126 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:02.023605 3008 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:02.024312 3036 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:02.024614 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:02.024971 31709 raft_consensus.cc:2354] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:25:02.025591 31594 raft_consensus.cc:2379] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 6: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:02.024346 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:02.027393 3007 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:02.046774 31597 raft_consensus.cc:1261] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260805 22:25:01.981479 3128 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
W20260805 22:25:02.053753 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 49933.1us
I20260805 22:25:02.045734 3125 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:02.061558 31600 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
W20260805 22:25:02.063043 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 216585us
W20260805 22:25:02.064795 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 84806.7us
I20260805 22:25:02.069309 31444 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:02.076697 31556 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:02.080263 3126 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:02.080982 2643 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260805 22:25:02.081061 3036 raft_consensus.cc:2726] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:02.085621 31556 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:02.089284 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:02.090046 31494 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
I20260805 22:25:02.091091 3036 raft_consensus.cc:2726] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:25:02.090855 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:02.091703 3125 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:02.070990 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:02.088037 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:02.096596 31602 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
I20260805 22:25:02.143658 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:02.145865 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 71633.2us
I20260805 22:25:02.148653 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:02.149456 31499 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:25:02.091897 32581 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:25:02.152741 31667 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:02.158195 31444 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:02.159909 31444 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:02.165655 32529 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:02.167793 32529 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:02.084257 3119 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:02.161966 2628 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:02.175258 2631 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:02.176344 1989 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260805 22:25:02.150105 3137 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:02.183493 31447 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:02.157320 3008 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:25:02.228802 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 67786.3us
I20260805 22:25:02.230299 31444 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:02.232578 31444 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:02.163523 2643 raft_consensus.cc:2726] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260805 22:25:02.233657 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 65942.6us
I20260805 22:25:02.170696 3050 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election won for term 8
W20260805 22:25:02.234520 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 51128.4us
I20260805 22:25:02.161540 2956 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:02.178282 2993 raft_consensus.cc:2781] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:02.157186 3126 consensus_queue.cc:1046] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.078s
I20260805 22:25:02.092453 2762 consensus_queue.cc:260] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:02.235934 3157 log.cc:924] Time spent T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.052s user 0.001s sys 0.000s
I20260805 22:25:02.414207 31706 raft_consensus.cc:1261] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260805 22:25:02.245678 2842 raft_consensus.cc:2781] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:02.415875 2842 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:02.416270 2842 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:02.225873 3121 raft_consensus.cc:2781] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260805 22:25:02.245841 3172 log.cc:924] Time spent T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.059s user 0.001s sys 0.000s
W20260805 22:25:02.257244 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 338512us
I20260805 22:25:02.235610 2945 raft_consensus.cc:2726] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260805 22:25:02.234289 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 290043us
I20260805 22:25:02.242735 3008 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:02.263047 31499 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:25:02.550990 31499 raft_consensus.cc:595] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:02.372087 32529 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:02.552299 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 180287us
W20260805 22:25:02.424451 3026 consensus_meta.cc:287] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.668s user 0.010s sys 0.002s
I20260805 22:25:02.172356 2944 consensus_queue.cc:1035] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.087s
I20260805 22:25:02.185137 2628 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:02.554914 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:25:02.375329 3010 consensus_meta.cc:287] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.588s user 0.010s sys 0.003s
I20260805 22:25:02.558995 32529 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:02.380347 2993 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260805 22:25:02.570158 32584 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.682s user 0.061s sys 0.069s
I20260805 22:25:02.421463 2971 raft_consensus.cc:2726] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260805 22:25:02.582254 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:02.591518 32579 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
W20260805 22:25:02.592532 32584 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.705s user 0.061s sys 0.069s
I20260805 22:25:02.460953 3121 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:02.584544 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:25:02.553083 2985 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.520s user 0.017s sys 0.033s
I20260805 22:25:02.461550 3025 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:02.589695 31707 raft_consensus.cc:2379] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 10: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:02.569268 1989 raft_consensus.cc:2726] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:02.377364 3007 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.200s
I20260805 22:25:02.634945 31709 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
W20260805 22:25:02.641433 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 477445us
I20260805 22:25:02.583969 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:02.568519 2643 raft_consensus.cc:3032] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 9 LEADER]: Stepping down as leader of term 9
W20260805 22:25:02.646071 32594 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.041s sys 0.042s
W20260805 22:25:02.596098 2975 consensus_meta.cc:287] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.747s user 0.005s sys 0.005s
I20260805 22:25:02.410275 2956 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:02.654505 32594 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.564s user 0.041s sys 0.042s
I20260805 22:25:02.660768 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:02.595541 2762 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
W20260805 22:25:02.640704 2985 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.605s user 0.018s sys 0.033s
I20260805 22:25:02.671846 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:02.677990 32579 raft_consensus.cc:3037] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 4
I20260805 22:25:02.632596 2993 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:02.639441 3121 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:02.637287 3042 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:02.647274 3010 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:02.636636 3173 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:02.578292 3026 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:02.660643 2643 raft_consensus.cc:729] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
W20260805 22:25:02.678735 2955 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.502s user 0.010s sys 0.019s
W20260805 22:25:02.714454 31714 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1891) took 1682 ms (client timeout 1685 ms). Trace:
W20260805 22:25:02.715088 31714 rpcz_store.cc:269] 0805 22:25:01.028905 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:01.032476 (+ 3571us) service_pool.cc:227] Handling call
0805 22:25:02.705342 (+1672866us) spinlock_profiling.cc:243] Waited 765 ms on lock 0x7b54000339d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:02.710957 (+ 5615us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1758659840}
W20260805 22:25:02.717796 31559 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.620s user 0.089s sys 0.161s
W20260805 22:25:02.719820 31559 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.794s user 0.090s sys 0.162s
W20260805 22:25:02.704779 2948 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.535s user 0.009s sys 0.028s
I20260805 22:25:02.651371 3025 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:02.705421 3010 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260805 22:25:02.719193 2955 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.538s user 0.010s sys 0.019s
W20260805 22:25:02.727396 2948 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.566s user 0.009s sys 0.029s
I20260805 22:25:02.753301 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:02.672372 2975 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:02.840268 31704 consensus_meta.cc:287] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.779s user 0.007s sys 0.005s
I20260805 22:25:02.842080 31704 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
I20260805 22:25:02.710561 3042 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:02.735148 3010 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Advancing to term 7
W20260805 22:25:02.838752 3167 log.cc:924] Time spent T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.654s user 0.000s sys 0.006s
I20260805 22:25:02.851408 32579 raft_consensus.cc:2454] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:25:02.833539 2955 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:02.839318 3098 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:25:02.845575 3168 log.cc:924] Time spent T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.198s user 0.000s sys 0.007s
I20260805 22:25:02.852814 3177 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:02.992373 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:02.872978 3036 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:02.995229 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:03.002625 31494 raft_consensus.cc:3037] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:02.873986 3137 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:02.872776 3119 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:02.891047 2628 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:02.992964 3050 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.586s user 0.007s sys 0.026s
I20260805 22:25:03.012382 31493 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40740 (request call id 1313) took 1263 ms. Trace:
W20260805 22:25:03.059948 31444 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.684s user 0.012s sys 0.032s
W20260805 22:25:03.060343 31444 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.685s user 0.012s sys 0.032s
I20260805 22:25:02.881979 2955 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:03.060916 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 828348us
I20260805 22:25:03.062295 31444 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:03.064888 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 505932us
I20260805 22:25:03.048039 31493 rpcz_store.cc:276] 0805 22:25:01.739027 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:01.739144 (+ 117us) service_pool.cc:227] Handling call
0805 22:25:01.739450 (+ 306us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:01.739655 (+ 205us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:01.739678 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:01.740087 (+ 409us) log.cc:842] Serialized 26 byte log entry
0805 22:25:02.061923 (+321836us) raft_consensus.cc:1631] Marking committed up to 7
0805 22:25:02.061960 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:02.062031 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:03.001752 (+939721us) raft_consensus.cc:1671] finished
0805 22:25:03.001771 (+ 19us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:03.001910 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":321523,"threads_started":1,"wal-append.queue_time_us":444325}
I20260805 22:25:03.073050 31556 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:02.841540 2643 consensus_queue.cc:260] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:03.011039 3080 consensus_meta.cc:287] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.964s user 0.007s sys 0.013s
W20260805 22:25:03.052716 3050 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.667s user 0.008s sys 0.027s
I20260805 22:25:03.093113 3050 raft_consensus.cc:686] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 8 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:03.082486 31444 leader_election.cc:398] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:25:03.093963 31444 leader_election.cc:401] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260805 22:25:03.095407 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:03.096092 31493 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 7
I20260805 22:25:03.006733 3055 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:03.104460 3050 consensus_queue.cc:237] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.105592 3131 raft_consensus.cc:2726] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:25:03.108570 31494 raft_consensus.cc:2454] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:25:03.108471 31444 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:03.114033 3179 log.cc:924] Time spent T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.106s user 0.000s sys 0.000s
W20260805 22:25:03.121160 31668 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.547s user 0.112s sys 0.197s
W20260805 22:25:03.126621 31668 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.553s user 0.112s sys 0.197s
I20260805 22:25:02.894797 3098 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.121148 3184 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:03.095268 2944 raft_consensus.cc:2726] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260805 22:25:03.016646 3137 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.136768 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:03.136969 32584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57366 (request call id 530) took 1373 ms. Trace:
W20260805 22:25:03.177450 3191 log.cc:924] Time spent T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.105s user 0.001s sys 0.000s
I20260805 22:25:03.180723 32584 rpcz_store.cc:276] 0805 22:25:01.763193 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:01.763300 (+ 107us) service_pool.cc:227] Handling call
0805 22:25:01.763555 (+ 255us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:01.821805 (+ 58250us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:01.821832 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:01.849696 (+ 27864us) log.cc:842] Serialized 26 byte log entry
0805 22:25:01.887691 (+ 37995us) spinlock_profiling.cc:243] Waited 4.91 ms on lock 0x7b18001bdfc0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e122345 00007f2736f59a39 00007f272e13f8bb 00007f272e13dcbf 00007f272e13fe22 00007f2730ce95bb 00007f2730cef063 00007f2730cef411 00007f273103c5cf
0805 22:25:02.073088 (+185397us) spinlock_profiling.cc:243] Waited 3.85 ms on lock 0x7b48000213d8. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2735f0f7f1 00007f2735f068e7 00007f2735f0677a 00007f2735ffa95f 00007f2735ffa627 00007f273600eb2c 00007f273600ead8 00007f273600ea58 00007f273600ea13 00007f273600dcbc 00007f2734f8fbe6
0805 22:25:02.592988 (+519900us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:25:02.593260 (+ 272us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:02.593338 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:03.136745 (+543407us) raft_consensus.cc:1671] finished
0805 22:25:03.136767 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:03.136904 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57875,"raft.queue_time_us":58398,"raft.run_cpu_time_us":4279,"raft.run_wall_time_us":258758,"spinlock_wait_cycles":20143232,"thread_start_us":705098,"threads_started":1,"wal-append.queue_time_us":1023031}
I20260805 22:25:03.182132 3137 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:03.182679 3137 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:03.183495 31705 raft_consensus.cc:1226] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 2. Current term is 3. Ops: []
I20260805 22:25:03.183504 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:03.184919 31711 raft_consensus.cc:3037] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
W20260805 22:25:03.185698 3187 log.cc:924] Time spent T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.171s user 0.000s sys 0.001s
I20260805 22:25:03.186077 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:03.180548 2944 raft_consensus.cc:2781] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:25:03.187836 2944 raft_consensus.cc:686] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:03.188669 2944 consensus_queue.cc:237] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.190513 32584 raft_consensus.cc:1261] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260805 22:25:03.177541 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:03.183101 3137 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.192106 31711 raft_consensus.cc:2454] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 7.
I20260805 22:25:03.193820 3137 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:03.194656 3137 raft_consensus.cc:2781] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:25:03.195128 3137 raft_consensus.cc:686] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:03.195979 3137 consensus_queue.cc:237] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.199433 32583 raft_consensus.cc:2379] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 6: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:03.198812 31356 catalog_manager.cc:5495] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:25:03.200367 31356 catalog_manager.cc:5495] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 4 to 8, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 8 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:03.203162 31357 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47946 (request call id 186) took 4341 ms. Trace:
W20260805 22:25:03.206103 3128 consensus_meta.cc:287] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 1.134s user 0.010s sys 0.001s
I20260805 22:25:03.152968 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:03.206715 31714 raft_consensus.cc:2454] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:25:03.152992 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:03.046424 3119 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.207726 2955 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:25:03.209286 3119 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:03.154377 31493 raft_consensus.cc:2454] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 7.
I20260805 22:25:03.013211 3036 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.136057 3044 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:03.212401 3044 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.156723 1989 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:03.214825 1989 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.161755 2975 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:03.162225 3184 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:03.168231 2842 consensus_meta.cc:287] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.752s user 0.009s sys 0.000s
W20260805 22:25:03.174234 3183 log.cc:924] Time spent T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.440s user 0.001s sys 0.000s
W20260805 22:25:03.174926 3189 log.cc:924] Time spent T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.327s user 0.000s sys 0.001s
I20260805 22:25:03.176095 3196 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:03.176584 3068 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:03.094832 3201 consensus_queue.cc:1035] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.099s
I20260805 22:25:03.203601 31356 catalog_manager.cc:5495] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 2 to 6, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 6 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:03.203583 31357 rpcz_store.cc:276] 0805 22:24:58.861637 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:58.882908 (+ 21271us) service_pool.cc:227] Handling call
0805 22:25:03.203071 (+4320163us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0805 22:25:00.774122 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0805 22:25:00.825156 (+ 51034us) write_op.cc:432] Acquiring schema lock in shared mode
0805 22:25:00.825189 (+ 33us) write_op.cc:435] Acquired schema lock
0805 22:25:00.825200 (+ 11us) tablet.cc:660] Decoding operations
0805 22:25:00.826118 (+ 918us) write_op.cc:620] Acquiring the partition lock for write op
0805 22:25:00.826177 (+ 59us) write_op.cc:641] Partition lock acquired for write op
0805 22:25:00.826200 (+ 23us) tablet.cc:683] Acquiring locks for 9 operations
0805 22:25:00.826860 (+ 660us) tablet.cc:699] Row locks acquired
0805 22:25:00.826883 (+ 23us) write_op.cc:260] PREPARE: finished
0805 22:25:00.827030 (+ 147us) write_op.cc:270] Start()
0805 22:25:00.827130 (+ 100us) write_op.cc:276] Timestamp: P: 1785968700826986 usec, L: 0
0805 22:25:00.827150 (+ 20us) op_driver.cc:331] REPLICATION: starting
0805 22:25:00.827730 (+ 580us) log.cc:842] Serialized 3601 byte log entry
0805 22:25:01.219576 (+391846us) op_driver.cc:444] REPLICATION: finished
0805 22:25:01.745663 (+526087us) write_op.cc:301] APPLY: starting
0805 22:25:01.745838 (+ 175us) tablet.cc:1365] starting BulkCheckPresence
0805 22:25:01.746270 (+ 432us) tablet.cc:1368] finished BulkCheckPresence
0805 22:25:01.746429 (+ 159us) tablet.cc:1370] starting ApplyRowOperation cycle
0805 22:25:01.798502 (+ 52073us) tablet.cc:1381] finished ApplyRowOperation cycle
0805 22:25:01.798737 (+ 235us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0805 22:25:01.798782 (+ 45us) write_op.cc:312] APPLY: finished
0805 22:25:01.799073 (+ 291us) log.cc:842] Serialized 71 byte log entry
0805 22:25:01.799238 (+ 165us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:25:01.799359 (+ 121us) write_op.cc:454] Released schema lock
0805 22:25:01.799404 (+ 45us) write_op.cc:341] FINISH: Updating metrics
0805 22:25:01.799717 (+ 313us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:25:01.799747 (+ 30us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":145522,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":470259,"apply.run_cpu_time_us":7197,"apply.run_wall_time_us":82264,"num_ops":9,"prepare.queue_time_us":202370,"prepare.run_cpu_time_us":7325,"prepare.run_wall_time_us":87003,"raft.queue_time_us":238273,"raft.run_cpu_time_us":11860,"raft.run_wall_time_us":411848,"replication_time_us":381731,"thread_start_us":562779,"threads_started":4,"wal-append.queue_time_us":66961,"wal-append.run_cpu_time_us":13260,"wal-append.run_wall_time_us":1931681}]]}
I20260805 22:25:03.144474 31609 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:25:03.005626 3177 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.118494 3047 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:03.214417 3044 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:03.341746 32586 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.614s user 0.000s sys 0.011s
W20260805 22:25:03.327877 31709 consensus_meta.cc:287] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.662s user 0.010s sys 0.000s
I20260805 22:25:03.133225 3026 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:03.131281 3080 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
W20260805 22:25:03.483786 31669 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.680s user 0.000s sys 0.017s
W20260805 22:25:03.506194 31669 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.817s user 0.001s sys 0.023s
W20260805 22:25:03.363330 3182 log.cc:924] Time spent T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.524s user 0.000s sys 0.002s
I20260805 22:25:03.489282 31709 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
W20260805 22:25:03.223351 2762 consensus_meta.cc:287] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.538s user 0.009s sys 0.001s
I20260805 22:25:03.151865 2643 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Advancing to term 10
W20260805 22:25:03.487635 32586 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.769s user 0.002s sys 0.011s
I20260805 22:25:03.498557 31356 catalog_manager.cc:5495] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
W20260805 22:25:03.519393 31561 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.797s user 0.010s sys 0.069s
W20260805 22:25:03.526087 31561 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.933s user 0.010s sys 0.070s
I20260805 22:25:03.219758 3119 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
W20260805 22:25:03.331033 3186 log.cc:924] Time spent T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.481s user 0.002s sys 0.000s
W20260805 22:25:03.309242 3121 consensus_meta.cc:287] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.595s user 0.002s sys 0.007s
I20260805 22:25:03.487993 2716 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:03.220633 3025 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:03.318346 2975 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:03.522950 31601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 2333) took 1540 ms. Trace:
I20260805 22:25:03.532625 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:03.133881 3055 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.221926 3007 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:03.537743 31704 raft_consensus.cc:2396] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf for term 7 because replica has last-logged OpId of term: 6 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260805 22:25:03.507030 3098 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:03.539026 31667 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.885s user 0.041s sys 0.055s
I20260805 22:25:03.531548 31356 catalog_manager.cc:5495] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 5 to 7, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 7 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:25:03.472422 3209 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:03.371276 3042 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:03.504818 2452 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:03.547214 32587 raft_consensus.cc:2454] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
W20260805 22:25:03.538524 3037 consensus_meta.cc:287] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 1.495s user 0.009s sys 0.006s
I20260805 22:25:03.525599 1989 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:03.548852 31667 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.895s user 0.041s sys 0.055s
W20260805 22:25:03.563175 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 909124us
I20260805 22:25:03.527261 3026 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:03.528388 3192 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:03.249800 3010 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.547610 31601 rpcz_store.cc:276] 0805 22:25:01.982304 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:02.631672 (+649368us) service_pool.cc:227] Handling call
0805 22:25:02.633982 (+ 2310us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:02.639136 (+ 5154us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:02.639167 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:02.639631 (+ 464us) log.cc:842] Serialized 26 byte log entry
0805 22:25:03.150836 (+511205us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:25:03.340710 (+189874us) log.cc:842] Serialized 14 byte log entry
0805 22:25:03.340979 (+ 269us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:03.341072 (+ 93us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:03.522707 (+181635us) raft_consensus.cc:1671] finished
0805 22:25:03.522729 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:03.522883 (+ 154us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":126874,"prepare.queue_time_us":228117,"prepare.run_cpu_time_us":36,"prepare.run_wall_time_us":37,"raft.queue_time_us":61868,"raft.run_cpu_time_us":313,"raft.run_wall_time_us":39586,"thread_start_us":478557,"threads_started":1,"wal-append.queue_time_us":832971}
I20260805 22:25:03.504479 3211 consensus_queue.cc:1035] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.044s
I20260805 22:25:03.533778 2955 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:03.580390 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:03.578912 31708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 2308) took 1606 ms. Trace:
W20260805 22:25:03.583794 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 509717us
I20260805 22:25:03.497617 2842 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.585588 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:03.586553 31602 raft_consensus.cc:2396] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
I20260805 22:25:03.503609 3196 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.583894 31708 rpcz_store.cc:276] 0805 22:25:01.972115 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:01.973494 (+ 1379us) service_pool.cc:227] Handling call
0805 22:25:01.974021 (+ 527us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:01.974386 (+ 365us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:01.974411 (+ 25us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:01.974909 (+ 498us) log.cc:842] Serialized 26 byte log entry
0805 22:25:02.378429 (+403520us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:25:02.378469 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:02.378555 (+ 86us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:03.578680 (+1200125us) raft_consensus.cc:1671] finished
0805 22:25:03.578701 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:03.578856 (+ 155us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":373,"raft.run_cpu_time_us":104,"raft.run_wall_time_us":106,"thread_start_us":403099,"threads_started":1,"wal-append.queue_time_us":713794}
I20260805 22:25:03.526999 3080 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.217832 3128 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.503774 3068 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.504045 3047 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.548434 3184 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:03.553253 2993 consensus_meta.cc:287] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.844s user 0.012s sys 0.000s
I20260805 22:25:03.582949 31601 raft_consensus.cc:2454] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:25:03.574316 31667 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:03.585211 31356 catalog_manager.cc:5495] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:03.548429 3036 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:03.580554 31608 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.948s user 0.018s sys 0.018s
I20260805 22:25:03.594390 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:03.638239 31450 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.012s sys 0.030s
W20260805 22:25:03.638695 31450 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.528s user 0.013s sys 0.030s
W20260805 22:25:03.639669 31444 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.530s user 0.013s sys 0.014s
W20260805 22:25:03.639971 31444 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.530s user 0.013s sys 0.014s
I20260805 22:25:03.608340 2762 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:03.548940 2716 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.577531 3192 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.567109 2452 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.563982 3007 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.582181 3203 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:03.590785 2955 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.589032 3194 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:03.594507 3144 raft_consensus.cc:3032] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Stepping down as leader of term 2
W20260805 22:25:03.657461 31447 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.009s sys 0.044s
I20260805 22:25:03.626506 3010 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:03.628005 2842 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:03.629676 3068 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:03.631294 3047 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:03.632073 32581 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
W20260805 22:25:03.634328 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 60120.5us
I20260805 22:25:03.539696 3119 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:03.640487 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 532068us
W20260805 22:25:03.638213 31608 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.047s user 0.018s sys 0.019s
I20260805 22:25:03.554117 2975 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.646296 31600 raft_consensus.cc:2379] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 7: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:03.544732 3121 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.653823 3197 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:03.656373 31500 raft_consensus.cc:2396] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 10 because replica has last-logged OpId of term: 9 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260805 22:25:03.572916 3037 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:03.663902 31447 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.530s user 0.010s sys 0.044s
I20260805 22:25:03.638801 3098 raft_consensus.cc:2726] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:25:03.640992 3184 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:03.640750 2993 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.579151 3026 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.672664 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:03.701894 31601 raft_consensus.cc:2379] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 10: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:03.664830 3144 raft_consensus.cc:729] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:03.675033 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:03.704209 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:03.704813 31601 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
W20260805 22:25:03.647073 31706 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.594s user 0.005s sys 0.028s
W20260805 22:25:03.684378 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 550893us
I20260805 22:25:03.565939 3209 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.687610 3098 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:03.688028 3184 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.696000 3208 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:03.699060 3037 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:03.701485 31356 catalog_manager.cc:5495] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 2 to 3, leader changed from 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 3 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
W20260805 22:25:03.707598 31706 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.654s user 0.006s sys 0.028s
I20260805 22:25:03.703663 31604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 2332) took 1721 ms. Trace:
I20260805 22:25:03.678714 2842 consensus_queue.cc:1035] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.035s
I20260805 22:25:03.722565 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:03.732937 31601 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:25:03.704620 31602 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
W20260805 22:25:03.727950 31670 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.942s user 0.024s sys 0.064s
W20260805 22:25:03.733716 31670 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.151s user 0.025s sys 0.064s
I20260805 22:25:03.733933 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:03.646915 2762 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.679140 3068 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:03.726817 32589 raft_consensus.cc:2379] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:03.731989 31706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1914) took 1181 ms. Trace:
I20260805 22:25:03.735405 31556 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:03.731552 31604 rpcz_store.cc:276] 0805 22:25:01.981987 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:03.175514 (+1193527us) service_pool.cc:227] Handling call
0805 22:25:03.175914 (+ 400us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:03.176152 (+ 238us) raft_consensus.cc:1520] Early marking committed up to index 6
0805 22:25:03.379337 (+203185us) log.cc:842] Serialized 14 byte log entry
0805 22:25:03.703282 (+323945us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:03.703347 (+ 65us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:25:03.703396 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:03.703463 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:03.703589 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":164418,"prepare.queue_time_us":211091,"prepare.run_cpu_time_us":34,"prepare.run_wall_time_us":34,"thread_start_us":323513,"threads_started":1}
I20260805 22:25:03.716929 3199 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:03.716182 3201 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:03.715380 3026 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:03.728276 3177 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:03.667569 3194 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.739416 31706 rpcz_store.cc:276] 0805 22:25:02.550341 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:02.550582 (+ 241us) service_pool.cc:227] Handling call
0805 22:25:02.583389 (+ 32807us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:02.583607 (+ 218us) raft_consensus.cc:1520] Early marking committed up to index 4
0805 22:25:03.053231 (+469624us) log.cc:842] Serialized 14 byte log entry
0805 22:25:03.731621 (+678390us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:03.731680 (+ 59us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:25:03.731731 (+ 51us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:03.731800 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:03.731928 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":836751,"prepare.run_cpu_time_us":50,"prepare.run_wall_time_us":6373,"raft.queue_time_us":1913,"thread_start_us":1087859,"threads_started":2}
I20260805 22:25:03.742904 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:03.713310 3205 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:03.715850 3209 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:03.729014 3098 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.746622 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:03.748374 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:03.764442 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:03.794051 31704 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:25:03.775796 31602 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:03.779627 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:03.663870 3203 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:03.742946 32529 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.195:44831 timed out after 1.502s (SENT) [suppressed 15 similar messages]
W20260805 22:25:03.795782 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 52669.5us
I20260805 22:25:03.794070 31601 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
I20260805 22:25:03.817443 31602 raft_consensus.cc:1261] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 8. (index mismatch)
I20260805 22:25:03.821012 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:03.821684 3121 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:03.821835 32583 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:03.701232 3197 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.822482 3184 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:03.770681 3229 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:03.771035 2953 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:03.824837 31444 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:03.831077 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:03.826005 31448 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.534s user 0.004s sys 0.024s
W20260805 22:25:03.832430 31448 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.723s user 0.004s sys 0.024s
I20260805 22:25:03.829272 31356 catalog_manager.cc:5495] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 4 to 5, leader changed from 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 5 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:25:03.783499 3037 raft_consensus.cc:2781] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:25:03.715945 3144 consensus_queue.cc:260] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.795045 3080 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:03.843086 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:03.770144 3068 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.889694 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:03.890650 31501 raft_consensus.cc:2379] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 4: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:03.734844 3208 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:03.892711 3208 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:03.846561 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 262789us
I20260805 22:25:03.896201 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:03.896849 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:03.897331 31554 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:25:03.897572 32594 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:03.765329 3201 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.901156 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:03.905893 31501 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 9.
I20260805 22:25:03.763653 3199 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.907532 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:03.904470 31356 catalog_manager.cc:5495] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:03.827502 3184 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:03.954092 31490 raft_consensus.cc:2379] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 8: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:03.954923 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:03.884878 2452 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:03.828272 3196 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:03.886683 3233 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:03.849701 3235 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:03.839525 3128 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:03.841455 3037 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:03.889539 3007 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:03.886760 3055 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:03.966367 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:03.968396 31706 raft_consensus.cc:1226] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 9. Current term is 10. Ops: [9.6-9.6]
I20260805 22:25:03.968624 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:03.889595 3203 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:03.971746 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 228817us
I20260805 22:25:03.972820 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:03.845072 3119 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:03.974519 31554 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:03.908466 3052 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:03.899840 3192 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:03.793910 3205 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.994393 32529 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:03.964416 2975 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:03.904829 3036 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:03.903884 2955 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:03.918797 3144 raft_consensus.cc:3037] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:25:03.976756 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:03.921200 3208 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:03.983240 31711 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:03.919180 2643 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:03.984108 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:25:03.984583 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 144699us
I20260805 22:25:03.981637 2842 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:03.975608 3272 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:03.985973 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:03.992524 31554 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:03.992786 31667 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:03.992154 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:03.964659 3184 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.838486 2953 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.975539 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:04.043498 31493 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:25:03.974083 2452 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:03.955684 3254 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:04.003507 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:04.020562 3196 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:03.970829 3026 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:04.014684 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:04.015360 2975 raft_consensus.cc:2781] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader election won for term 7
W20260805 22:25:04.017045 3236 log.cc:924] Time spent T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.190s user 0.000s sys 0.000s
I20260805 22:25:04.017799 32529 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:25:04.061254 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 43540.8us
I20260805 22:25:04.062093 32529 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:04.018226 3268 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:03.839179 3229 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.976976 3037 raft_consensus.cc:3037] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:03.973625 2993 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:03.974362 3235 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:04.023615 31715 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:25:04.049801 3080 raft_consensus.cc:2781] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:25:03.985037 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:25:04.017804 3222 log.cc:924] Time spent T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.191s user 0.000s sys 0.001s
I20260805 22:25:03.978070 3125 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:03.985216 31501 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:25:04.031394 31711 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:25:04.028294 3208 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.997942 2762 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:04.036736 2842 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:04.037200 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 44774.8us
I20260805 22:25:04.036942 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:04.000830 3055 raft_consensus.cc:2726] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:25:04.038196 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:04.006764 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:04.011343 3197 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:04.054963 31495 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:04.059671 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:04.074219 3194 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:04.070540 31709 raft_consensus.cc:2379] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 8: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:03.973407 3233 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:04.021920 3036 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:04.030663 2643 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:03.994232 3052 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:04.035919 31667 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:04.071637 3203 raft_consensus.cc:2781] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election won for term 4
W20260805 22:25:04.117720 3182 log.cc:924] Time spent T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260805 22:25:04.044234 3272 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:04.254755 2762 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:04.074429 3098 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:04.255031 3270 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260805 22:25:04.180291 32594 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:25:04.255451 3270 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:04.255805 3270 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:04.067531 32529 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:04.106184 2975 raft_consensus.cc:686] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [term 7 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:04.256209 3272 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:04.257472 3232 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:04.117771 3209 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:04.212736 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:04.254060 3271 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:04.057132 2452 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:04.258268 3271 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:04.259846 2452 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:04.260324 3271 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:04.205783 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 195692us
W20260805 22:25:04.185436 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 288460us
I20260805 22:25:04.261417 2762 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:04.261857 2762 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:04.099447 3268 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:04.266395 3205 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:04.266655 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:04.266911 3209 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:04.256155 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:04.268486 32594 raft_consensus.cc:2379] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 7: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:04.268774 3209 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:04.269184 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:25:04.270121 3265 log.cc:924] Time spent T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.229s user 0.001s sys 0.000s
I20260805 22:25:04.270902 2842 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:04.146337 3128 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:04.072757 3026 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:04.125195 3235 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:04.069407 3196 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:04.312661 3196 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:04.293964 3007 raft_consensus.cc:2781] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:25:04.316876 3007 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:04.317258 3007 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:04.317677 3251 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:04.318284 3251 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:04.293625 3232 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:04.320478 3251 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:04.322739 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:04.249898 3194 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:04.177130 3142 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:04.162029 3080 raft_consensus.cc:686] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 6 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:04.338202 3208 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:04.253618 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 217767us
I20260805 22:25:04.339118 31667 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:04.339984 31667 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:04.341017 31667 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:04.254017 3203 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:04.341569 3203 raft_consensus.cc:3037] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:04.257025 2975 consensus_queue.cc:237] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:04.343429 32579 raft_consensus.cc:2454] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:25:04.184490 32583 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:25:04.267218 3268 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:04.271453 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 203992us
I20260805 22:25:04.273931 31602 raft_consensus.cc:2379] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 7: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:04.244905 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:04.164021 3238 raft_consensus.cc:2726] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260805 22:25:04.177474 3010 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:04.213352 3252 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:04.160373 3125 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:04.217473 2956 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:04.299579 2953 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:04.301699 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:04.306005 3026 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:04.293711 31668 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:04.161540 3119 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:04.186571 3068 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:04.322244 31494 raft_consensus.cc:1261] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260805 22:25:04.327809 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:04.329859 31495 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 7.
I20260805 22:25:04.332495 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:04.265141 3052 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:04.323383 31493 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:25:04.337780 3080 consensus_queue.cc:237] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:04.069115 3254 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:04.341660 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:04.322343 3232 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:04.295538 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:04.294184 3128 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:04.713930 3238 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:04.714569 3238 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:04.716668 3238 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:04.718696 3008 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:04.385890 3037 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:04.524792 3184 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:04.721652 3252 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:04.724972 31490 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:25:04.523694 3144 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:04.729099 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:04.391016 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:04.421048 3281 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:04.638137 3270 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:04.709067 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:04.620862 3244 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:04.430075 2452 consensus_queue.cc:1035] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.308s
I20260805 22:25:04.576175 2993 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:04.408977 3010 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:04.753885 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 460242us
I20260805 22:25:04.754348 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:04.755896 3098 raft_consensus.cc:2781] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:04.756469 3098 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260805 22:25:04.708595 3178 log.cc:924] Time spent T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 1.865s user 0.009s sys 0.002s
I20260805 22:25:04.757079 3098 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:04.475185 3271 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:04.758625 32591 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57366 (request call id 531) took 2995 ms. Trace:
I20260805 22:25:04.758746 3271 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:04.759298 32591 rpcz_store.cc:276] 0805 22:25:01.763500 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:01.763810 (+ 310us) service_pool.cc:227] Handling call
0805 22:25:01.764099 (+ 289us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:01.822513 (+ 58414us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:01.822542 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:01.840371 (+ 17829us) log.cc:842] Serialized 26 byte log entry
0805 22:25:02.226752 (+386381us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:25:02.226801 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:02.226889 (+ 88us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:04.708009 (+2481120us) spinlock_profiling.cc:243] Waited 7.14 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:25:04.758132 (+ 50123us) raft_consensus.cc:1671] finished
0805 22:25:04.758155 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:04.758510 (+ 355us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58023,"raft.queue_time_us":58135,"raft.run_cpu_time_us":165,"raft.run_wall_time_us":27329,"spinlock_wait_cycles":16416128,"thread_start_us":337130,"threads_started":1,"wal-append.queue_time_us":816637}
W20260805 22:25:04.768791 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 51014.2us
I20260805 22:25:04.755442 31490 raft_consensus.cc:2379] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 9: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:04.773914 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:04.774724 3128 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:04.775720 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:04.776891 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:04.760727 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:04.778227 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:04.779557 3026 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:04.789929 3026 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:04.794086 3026 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:04.785395 3077 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:04.795994 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:04.729614 3144 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:04.369076 3272 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:04.799767 3272 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:04.800587 3272 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:04.801309 3144 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:04.780539 2956 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:04.802484 2956 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:04.805855 2956 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:04.411590 31423 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica f8e9f4c756da4203917c464e2b179994 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:04.808300 3144 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:04.795303 3077 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:04.786803 31667 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:04.815080 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:04.820367 3055 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:04.347988 3142 raft_consensus.cc:3037] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:04.826171 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
W20260805 22:25:04.838651 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 51963.1us
I20260805 22:25:04.410549 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:04.869520 31502 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:25:04.840848 31492 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 9.
I20260805 22:25:04.841452 31486 raft_consensus.cc:2354] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 6. Current term is 7.
W20260805 22:25:04.841833 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 71866.5us
I20260805 22:25:04.843160 31706 raft_consensus.cc:2379] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:04.823371 3010 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:04.334856 3177 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:04.844754 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:04.876876 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:04.721158 3037 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:04.864720 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:25:04.827590 3235 consensus_meta.cc:287] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.513s user 0.007s sys 0.003s
I20260805 22:25:04.849177 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:04.891146 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:04.825151 2993 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:04.895421 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:04.927486 31706 raft_consensus.cc:2354] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 9. Current term is 10.
I20260805 22:25:04.928577 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:04.900990 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:04.929684 31486 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:25:04.940724 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:04.929284 31706 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:25:04.962766 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:04.971359 31495 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 8.
I20260805 22:25:04.985690 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:04.986691 31490 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:25:05.004580 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:05.005360 31492 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:05.007119 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.007879 31604 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:25:05.009668 31713 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:25:04.394395 3268 raft_consensus.cc:2781] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:05.011684 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:04.866084 2452 raft_consensus.cc:2726] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260805 22:25:05.012877 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:25:05.012657 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 124062us
I20260805 22:25:05.016934 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:05.019568 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.020313 31604 raft_consensus.cc:2379] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 7: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:04.599749 3229 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:05.016345 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:04.420792 3201 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:04.345937 3194 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:04.422307 2762 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:04.849821 3144 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:04.924667 3235 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:04.882552 3055 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:04.866546 3119 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:04.827035 3244 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:04.825692 3281 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:05.412210 31494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1889) took 1025 ms. Trace:
I20260805 22:25:05.181205 32529 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:05.122315 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:25:05.418398 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 237248us
I20260805 22:25:05.412312 31486 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1887) took 1026 ms. Trace:
I20260805 22:25:05.419059 31486 rpcz_store.cc:276] 0805 22:25:04.386160 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.019148 (+632988us) service_pool.cc:227] Handling call
0805 22:25:05.024036 (+ 4888us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:05.024199 (+ 163us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:05.024225 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:05.024264 (+ 39us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:25:05.024295 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:05.024342 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:05.412254 (+387912us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:05.418701 31706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1925) took 1082 ms. Trace:
I20260805 22:25:05.418045 31494 rpcz_store.cc:276] 0805 22:25:04.386768 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.020980 (+634212us) service_pool.cc:227] Handling call
0805 22:25:05.411754 (+390774us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:05.411926 (+ 172us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:05.411948 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:05.411985 (+ 37us) raft_consensus.cc:1631] Marking committed up to 3
0805 22:25:05.412011 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:05.412057 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:05.412153 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:05.121213 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.179816 31554 leader_election.cc:398] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 9. Current term is 10.
I20260805 22:25:05.421037 31706 rpcz_store.cc:276] 0805 22:25:04.336730 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.121964 (+785234us) service_pool.cc:227] Handling call
0805 22:25:05.418638 (+296674us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:05.032359 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:04.893469 32585 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:25:05.421742 31554 leader_election.cc:401] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
W20260805 22:25:05.422350 32585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57372 (request call id 693) took 1406 ms (client timeout 1686 ms). Trace:
I20260805 22:25:05.423385 31704 raft_consensus.cc:1261] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260805 22:25:05.211113 31492 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:25:05.426333 31492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1897) took 1037 ms. Trace:
I20260805 22:25:05.426635 31492 rpcz_store.cc:276] 0805 22:25:04.388847 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:04.870477 (+481630us) service_pool.cc:227] Handling call
0805 22:25:05.426276 (+555799us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:04.875703 3252 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:05.428907 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:05.429787 31494 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
W20260805 22:25:05.424269 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 244594us
I20260805 22:25:05.431046 31554 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:05.421533 31604 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:25:05.422513 31597 raft_consensus.cc:2396] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 8 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260805 22:25:05.422783 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:05.014370 3268 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
W20260805 22:25:05.404358 3290 log.cc:924] Time spent T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.763s user 0.002s sys 0.000s
I20260805 22:25:05.446671 31502 raft_consensus.cc:1261] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260805 22:25:05.439752 31493 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 8.
I20260805 22:25:05.439855 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.442371 32529 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:05.437920 31492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1890) took 1050 ms. Trace:
W20260805 22:25:05.326774 3282 log.cc:924] Time spent T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.729s user 0.000s sys 0.002s
I20260805 22:25:05.428936 3254 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:05.390084 3270 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:04.912367 3177 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:05.438927 3229 raft_consensus.cc:2781] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:05.459641 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.123528 3008 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:05.410980 3303 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
W20260805 22:25:05.467867 31448 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.000s user 0.011s sys 0.014s
W20260805 22:25:05.468225 31448 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.042s user 0.011s sys 0.014s
I20260805 22:25:05.460505 31604 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:05.463960 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:05.460795 31492 rpcz_store.cc:276] 0805 22:25:04.387023 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.437013 (+1049990us) service_pool.cc:227] Handling call
0805 22:25:05.437366 (+ 353us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:05.437573 (+ 207us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:05.437600 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:05.437650 (+ 50us) raft_consensus.cc:1631] Marking committed up to 3
0805 22:25:05.437695 (+ 45us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:05.437747 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:05.437859 (+ 112us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260805 22:25:05.464855 31444 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.560s user 0.012s sys 0.047s
I20260805 22:25:05.464912 31706 raft_consensus.cc:2379] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 9: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:05.428586 3286 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Advancing to term 10
I20260805 22:25:05.021662 2975 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
W20260805 22:25:05.458691 3137 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 2.260s user 0.020s sys 0.035s
I20260805 22:25:05.095451 2716 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:05.458309 3098 consensus_meta.cc:287] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.700s user 0.008s sys 0.006s
I20260805 22:25:05.470357 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:05.469520 31444 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.565s user 0.012s sys 0.047s
I20260805 22:25:05.438179 3238 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:05.447300 3268 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:05.457008 3271 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:05.480155 31604 raft_consensus.cc:2396] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf for term 7 because replica has last-logged OpId of term: 5 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 6.
W20260805 22:25:05.478840 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 574208us
I20260805 22:25:05.460096 3125 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:05.462807 3229 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:05.459892 2993 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:05.493785 2993 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:25:05.494544 2993 raft_consensus.cc:686] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 7 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:05.495426 2993 consensus_queue.cc:237] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 5.8, Last appended by leader: 8, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:05.497973 3300 log.cc:924] Time spent T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.179s user 0.001s sys 0.000s
I20260805 22:25:05.486353 3238 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:05.499402 31594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 2352) took 1472 ms. Trace:
I20260805 22:25:05.499763 31594 rpcz_store.cc:276] 0805 22:25:04.026799 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:04.261981 (+235182us) service_pool.cc:227] Handling call
0805 22:25:04.262240 (+ 259us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:04.723581 (+461341us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:04.723613 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:04.723652 (+ 39us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:25:04.723681 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:05.499167 (+775486us) spinlock_profiling.cc:243] Waited 461 ms on lock 0x7b54003d59d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:05.499216 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:05.499344 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1060538112}
I20260805 22:25:05.500545 3238 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:05.471771 3077 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:05.476708 3303 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:05.502588 3055 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:05.503229 3303 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:05.507193 31715 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1922) took 1218 ms. Trace:
I20260805 22:25:05.507591 31715 rpcz_store.cc:276] 0805 22:25:04.288991 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:04.290112 (+ 1121us) service_pool.cc:227] Handling call
0805 22:25:04.290446 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:04.325729 (+ 35283us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:04.325759 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:04.325798 (+ 39us) raft_consensus.cc:1631] Marking committed up to 3
0805 22:25:04.325826 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:05.505298 (+1179472us) spinlock_profiling.cc:243] Waited 35.1 ms on lock 0x7b540016ce54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:05.505343 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:05.507084 (+ 1741us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":80647424}
I20260805 22:25:05.509270 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:05.510262 31715 raft_consensus.cc:2454] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 1.
I20260805 22:25:05.510689 31715 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1931) took 1124 ms. Trace:
I20260805 22:25:05.511198 31714 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 2322) took 1541 ms. Trace:
I20260805 22:25:05.482363 31486 raft_consensus.cc:1261] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 3. (index mismatch)
W20260805 22:25:05.422803 32585 rpcz_store.cc:269] 0805 22:25:04.016008 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:04.248536 (+232528us) service_pool.cc:227] Handling call
0805 22:25:05.422291 (+1173755us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:05.483094 31444 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:05.515659 3281 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:05.516379 31444 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:05.519246 2840 raft_consensus.cc:2781] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:25:05.519781 2840 raft_consensus.cc:686] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
W20260805 22:25:05.476243 3137 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 2.274s user 0.020s sys 0.035s
I20260805 22:25:05.520666 2840 consensus_queue.cc:237] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:05.524375 3271 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:05.524777 3286 raft_consensus.cc:2726] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 9. Current term is 10.
I20260805 22:25:05.483421 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:05.488198 3296 log.cc:924] Time spent T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.199s user 0.000s sys 0.001s
I20260805 22:25:05.488376 3119 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:05.488588 3229 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
W20260805 22:25:05.489003 3304 log.cc:924] Time spent T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.099s user 0.000s sys 0.001s
I20260805 22:25:05.489871 3177 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:05.490028 3235 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:05.490834 3298 raft_consensus.cc:2781] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:25:05.558137 3298 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:05.524924 3271 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:05.490981 3244 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:05.491454 3306 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:05.559486 3306 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:05.560592 3271 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:05.561910 3306 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:05.475780 2975 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:05.563966 2975 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:05.482543 3098 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:05.565896 3098 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:05.501154 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.510999 31715 rpcz_store.cc:276] 0805 22:25:04.386343 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.508942 (+1122599us) service_pool.cc:227] Handling call
0805 22:25:05.510638 (+ 1696us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:05.568272 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:25:05.568825 3203 consensus_meta.cc:287] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 1.227s user 0.008s sys 0.001s
I20260805 22:25:05.569067 31715 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1926) took 1221 ms. Trace:
I20260805 22:25:05.569485 31715 rpcz_store.cc:276] 0805 22:25:04.347227 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.567992 (+1220765us) service_pool.cc:227] Handling call
0805 22:25:05.569009 (+ 1017us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:05.514513 2628 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:05.525859 31444 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:05.524778 31495 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1895) took 1136 ms. Trace:
I20260805 22:25:05.547513 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.581875 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.582561 31604 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
I20260805 22:25:05.583038 31604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59028 (request call id 2360) took 1232 ms. Trace:
I20260805 22:25:05.583321 31604 rpcz_store.cc:276] 0805 22:25:04.350701 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.581588 (+1230887us) service_pool.cc:227] Handling call
0805 22:25:05.582970 (+ 1382us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:05.511567 31714 rpcz_store.cc:276] 0805 22:25:03.969755 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:03.969906 (+ 151us) service_pool.cc:227] Handling call
0805 22:25:03.970263 (+ 357us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:03.971714 (+ 1451us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:03.971746 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:03.977504 (+ 5758us) log.cc:842] Serialized 26 byte log entry
0805 22:25:04.150415 (+172911us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:25:04.150453 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:04.150523 (+ 70us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:05.511001 (+1360478us) raft_consensus.cc:1671] finished
0805 22:25:05.511023 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:05.511145 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12322,"raft.queue_time_us":51011,"raft.run_cpu_time_us":150,"raft.run_wall_time_us":7891,"thread_start_us":130464,"threads_started":1,"wal-append.queue_time_us":491680}
I20260805 22:25:05.558617 3298 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:05.472046 3008 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:05.567334 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.581020 31495 rpcz_store.cc:276] 0805 22:25:04.388329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:04.876602 (+488273us) service_pool.cc:227] Handling call
0805 22:25:05.524615 (+648013us) spinlock_profiling.cc:243] Waited 7.76 ms on lock 0x7b54000f78d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:05.524720 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17853696}
I20260805 22:25:05.585335 31594 raft_consensus.cc:2379] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 4: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:05.588312 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:05.589195 31714 raft_consensus.cc:2379] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 6: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:05.592211 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:05.593434 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:05.591028 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.594769 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:05.595497 31715 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1935) took 1208 ms. Trace:
I20260805 22:25:05.595652 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:05.596508 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 70710.5us
I20260805 22:25:05.598527 31715 rpcz_store.cc:276] 0805 22:25:04.387419 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.570089 (+1182670us) service_pool.cc:227] Handling call
0805 22:25:05.595430 (+ 25341us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:05.599813 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:05.600653 31714 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1927) took 1247 ms. Trace:
I20260805 22:25:05.600970 31714 rpcz_store.cc:276] 0805 22:25:04.353335 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.599554 (+1246219us) service_pool.cc:227] Handling call
0805 22:25:05.600594 (+ 1040us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:05.601984 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:05.603229 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.602567 31714 raft_consensus.cc:2362] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7. Re-sending same reply.
I20260805 22:25:05.603617 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.604291 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:05.604862 31714 raft_consensus.cc:3037] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:05.615914 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.617462 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.618213 31594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59028 (request call id 2355) took 1284 ms. Trace:
I20260805 22:25:05.618546 31594 rpcz_store.cc:276] 0805 22:25:04.333612 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.617209 (+1283597us) service_pool.cc:227] Handling call
0805 22:25:05.618161 (+ 952us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260805 22:25:05.619228 3052 consensus_meta.cc:287] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.858s user 0.013s sys 0.000s
I20260805 22:25:05.619396 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.619679 3052 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:05.620501 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.621896 3052 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:05.621871 31597 raft_consensus.cc:2379] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 8: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:05.624290 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:05.624915 31715 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 8.
I20260805 22:25:05.615942 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.628391 32529 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:05.619465 3007 consensus_meta.cc:287] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 1.302s user 0.008s sys 0.005s
I20260805 22:25:05.584573 2628 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:05.614720 3196 raft_consensus.cc:2726] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:25:05.621129 31594 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 3.
I20260805 22:25:05.643755 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.584899 3203 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:05.645932 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:25:05.620281 32581 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57366 (request call id 541) took 2435 ms (client timeout 1581 ms). Trace:
I20260805 22:25:05.648965 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:05.648313 31715 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:25:05.688535 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.632033 2975 raft_consensus.cc:2726] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:25:05.714145 31594 raft_consensus.cc:2379] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 7: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:05.779634 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:05.780189 31490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1885) took 1429 ms. Trace:
I20260805 22:25:05.780678 31490 rpcz_store.cc:276] 0805 22:25:04.351145 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:04.987633 (+636488us) service_pool.cc:227] Handling call
0805 22:25:04.987952 (+ 319us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:04.988160 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 5
0805 22:25:05.471142 (+482982us) log.cc:842] Serialized 14 byte log entry
0805 22:25:05.779817 (+308675us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:05.779875 (+ 58us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:25:05.779924 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:05.779995 (+ 71us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:05.780123 (+ 128us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":636426,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":37,"thread_start_us":577885,"threads_started":2}
I20260805 22:25:05.645493 2762 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:05.799860 31501 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40740 (request call id 1326) took 1834 ms (client timeout 1735 ms). Trace:
W20260805 22:25:05.820209 31501 rpcz_store.cc:269] 0805 22:25:03.965869 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:03.966096 (+ 227us) service_pool.cc:227] Handling call
0805 22:25:05.799660 (+1833564us) spinlock_profiling.cc:243] Waited 3.53 ms on lock 0x7b5400121e54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:05.799780 (+ 120us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":8110336}
I20260805 22:25:05.821352 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:25:05.825464 31609 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59028 (request call id 2338) took 3818 ms (client timeout 1721 ms). Trace:
W20260805 22:25:05.826484 31600 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56384 (request call id 393) took 2221 ms (client timeout 1526 ms). Trace:
I20260805 22:25:05.826071 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:05.827750 31715 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:25:05.828603 31715 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58348 (request call id 597) took 1072 ms. Trace:
I20260805 22:25:05.829219 31715 rpcz_store.cc:276] 0805 22:25:04.756508 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.733017 (+976509us) service_pool.cc:227] Handling call
0805 22:25:05.828546 (+ 95529us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:05.804638 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.831745 31594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59028 (request call id 2361) took 1439 ms. Trace:
I20260805 22:25:05.832075 31594 rpcz_store.cc:276] 0805 22:25:04.392596 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.795259 (+1402663us) service_pool.cc:227] Handling call
0805 22:25:05.831682 (+ 36423us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260805 22:25:05.800761 31493 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1896) took 1412 ms (client timeout 1615 ms). Trace:
I20260805 22:25:05.803875 31499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:54616 (request call id 19) took 3518 ms. Trace:
I20260805 22:25:05.806560 31502 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1888) took 1420 ms. Trace:
I20260805 22:25:05.806849 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:25:05.826877 31600 rpcz_store.cc:269] 0805 22:25:03.580330 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:03.580495 (+ 165us) service_pool.cc:227] Handling call
0805 22:25:05.801301 (+2220806us) spinlock_profiling.cc:243] Waited 7.69 ms on lock 0x7b540005d7d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:05.801457 (+ 156us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":17698176}
W20260805 22:25:05.825969 31609 rpcz_store.cc:269] 0805 22:25:01.985134 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:02.003380 (+ 18246us) service_pool.cc:227] Handling call
0805 22:25:05.803508 (+3800128us) spinlock_profiling.cc:243] Waited 1.1 s on lock 0x7b5400005d54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:05.803621 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2524627712}
I20260805 22:25:05.838617 31603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 2345) took 2277 ms. Trace:
I20260805 22:25:05.664844 3194 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:05.841436 31502 rpcz_store.cc:276] 0805 22:25:04.386505 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.019486 (+632981us) service_pool.cc:227] Handling call
0805 22:25:05.128280 (+108794us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:05.447073 (+318793us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:05.806315 (+359242us) spinlock_profiling.cc:243] Waited 318 ms on lock 0x7b5400121454. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:05.806504 (+ 189us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":731399040}
I20260805 22:25:05.843416 31603 rpcz_store.cc:276] 0805 22:25:03.528451 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:03.529392 (+ 941us) service_pool.cc:227] Handling call
0805 22:25:03.545344 (+ 15952us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:03.715059 (+169715us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:03.715091 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:03.715141 (+ 50us) raft_consensus.cc:1631] Marking committed up to 1
0805 22:25:03.715170 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:05.805298 (+2090128us) spinlock_profiling.cc:243] Waited 169 ms on lock 0x7b54002382d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:05.805346 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:05.805486 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":389831936}
I20260805 22:25:05.839974 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:05.802593 31713 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56260 (request call id 2321) took 1833 ms (client timeout 1658 ms). Trace:
W20260805 22:25:05.847805 31713 rpcz_store.cc:269] 0805 22:25:03.969325 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:03.969466 (+ 141us) service_pool.cc:227] Handling call
0805 22:25:05.801842 (+1832376us) spinlock_profiling.cc:243] Waited 985 ms on lock 0x7b540016c954. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:05.802514 (+ 672us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2265286016}
W20260805 22:25:05.848618 31493 rpcz_store.cc:269] 0805 22:25:04.388595 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:04.760472 (+371877us) service_pool.cc:227] Handling call
0805 22:25:05.800597 (+1040125us) spinlock_profiling.cc:243] Waited 661 ms on lock 0x7b54002c24d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:05.800702 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1521100160}
I20260805 22:25:05.850097 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:05.850864 31493 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:25:05.775450 3119 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:05.773554 3010 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
W20260805 22:25:05.857165 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 172624us
I20260805 22:25:05.775789 3306 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:05.640154 3007 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:05.840716 31499 rpcz_store.cc:276] 0805 22:25:02.260092 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:02.260532 (+ 440us) service_pool.cc:227] Handling call
0805 22:25:05.777948 (+3517416us) spinlock_profiling.cc:243] Waited 5 ms on lock 0x7b5400308020. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2730ffddc0 00007f27310629c2 00007f2735fac81a 00007f272f69fd04 00007f272f69fc92 00007f272f69fbc1 00007f272f69fb3c 00007f272f69edf2 00007f272f49294c 00007f272f491d86
0805 22:25:05.778210 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11504512}
I20260805 22:25:05.861420 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:05.862779 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:25:05.894903 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 481749us
I20260805 22:25:05.866516 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:05.796156 3144 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:05.868052 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:05.804273 3052 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:05.864143 31604 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 10.
I20260805 22:25:05.795708 3303 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:05.897058 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:05.898964 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:05.897647 31715 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56260 (request call id 2329) took 1112 ms. Trace:
I20260805 22:25:05.843758 3308 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:05.916894 31494 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1902) took 1523 ms. Trace:
I20260805 22:25:05.945587 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:05.950690 31501 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:25:05.949692 31494 rpcz_store.cc:276] 0805 22:25:04.393344 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.439215 (+1045871us) service_pool.cc:227] Handling call
0805 22:25:05.439530 (+ 315us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:05.439721 (+ 191us) raft_consensus.cc:1520] Early marking committed up to index 6
0805 22:25:05.677167 (+237446us) log.cc:842] Serialized 14 byte log entry
0805 22:25:05.916548 (+239381us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:05.916604 (+ 56us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:25:05.916646 (+ 42us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:05.916714 (+ 68us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:05.916833 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":47143,"prepare.queue_time_us":270939,"prepare.run_cpu_time_us":42,"prepare.run_wall_time_us":42,"thread_start_us":405752,"threads_started":2}
I20260805 22:25:05.946303 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:05.952680 31444 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:05.960896 31495 raft_consensus.cc:2379] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:05.955096 3008 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:05.650911 32581 rpcz_store.cc:269] 0805 22:25:03.184481 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:03.185806 (+ 1325us) service_pool.cc:227] Handling call
0805 22:25:05.620083 (+2434277us) spinlock_profiling.cc:243] Waited 30 ms on lock 0x7b5400363dd4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:05.620193 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":68981120}
I20260805 22:25:05.968698 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.959882 31714 raft_consensus.cc:2454] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
W20260805 22:25:05.985685 31714 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1930) took 1599 ms (client timeout 2110 ms). Trace:
W20260805 22:25:05.986179 31714 rpcz_store.cc:269] 0805 22:25:04.386080 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.604087 (+1218007us) service_pool.cc:227] Handling call
0805 22:25:05.985621 (+381534us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:05.987145 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:05.987931 31714 raft_consensus.cc:2379] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 7: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:05.988341 31714 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56260 (request call id 2333) took 1201 ms. Trace:
I20260805 22:25:05.984465 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.990026 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.990753 31594 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:25:05.918076 31502 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:05.857640 3309 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:05.889204 2762 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:05.887962 3196 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:05.868227 3131 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:05.914286 3010 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:05.968709 3203 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:05.968982 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:06.022061 31603 raft_consensus.cc:2454] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
W20260805 22:25:05.970682 32589 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57372 (request call id 689) took 3416 ms (client timeout 1542 ms). Trace:
I20260805 22:25:05.971715 3007 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:05.973140 31711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1919) took 1721 ms. Trace:
I20260805 22:25:06.026311 31711 rpcz_store.cc:276] 0805 22:25:04.251504 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:04.251640 (+ 136us) service_pool.cc:227] Handling call
0805 22:25:04.251968 (+ 328us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:04.256942 (+ 4974us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:04.256969 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:04.257005 (+ 36us) raft_consensus.cc:1631] Marking committed up to 3
0805 22:25:04.257037 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:05.972929 (+1715892us) spinlock_profiling.cc:243] Waited 4.79 ms on lock 0x7b54002b2fd4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:05.972968 (+ 39us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:05.973086 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":11026688}
I20260805 22:25:06.027699 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:05.949190 3052 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:05.978020 31704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1921) took 1724 ms. Trace:
W20260805 22:25:05.977066 3273 log.cc:924] Time spent T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 1.727s user 0.004s sys 0.013s
I20260805 22:25:05.981087 3229 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:05.993738 3008 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:06.038066 31486 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:25:05.982373 3317 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:05.983070 3221 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:05.952527 3303 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:05.985049 31601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1300) took 2313 ms. Trace:
I20260805 22:25:05.949218 31715 rpcz_store.cc:276] 0805 22:25:04.785335 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.829894 (+1044559us) service_pool.cc:227] Handling call
0805 22:25:05.897586 (+ 67692us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:05.991590 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.984431 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:05.988691 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:25:06.048764 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 99376.6us
I20260805 22:25:06.048983 31495 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 9.
I20260805 22:25:06.003031 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:06.045342 31601 rpcz_store.cc:276] 0805 22:25:03.671776 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:03.797076 (+125300us) service_pool.cc:227] Handling call
0805 22:25:03.797437 (+ 361us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:03.928318 (+130881us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:03.928353 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:03.928395 (+ 42us) raft_consensus.cc:1631] Marking committed up to 7
0805 22:25:03.928426 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:05.984810 (+2056384us) spinlock_profiling.cc:243] Waited 131 ms on lock 0x7b5400239bd4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:05.984851 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:05.984985 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":300476672}
I20260805 22:25:05.944682 2628 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:06.022868 32589 rpcz_store.cc:269] 0805 22:25:02.554195 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:02.554779 (+ 584us) service_pool.cc:227] Handling call
0805 22:25:02.753133 (+198354us) spinlock_profiling.cc:243] Waited 3.04 ms on lock 0x7b5800005198. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735faa1b0 00007f2735fa9a94 00007f272f691c04 00007f272f691b92
0805 22:25:05.970474 (+3217341us) spinlock_profiling.cc:243] Waited 895 ms on lock 0x7b54004a1854. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:05.970612 (+ 138us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2065845248}
W20260805 22:25:05.995837 3272 consensus_meta.cc:287] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 1.194s user 0.008s sys 0.006s
I20260805 22:25:06.051602 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
W20260805 22:25:06.049881 31709 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56260 (request call id 2320) took 2081 ms (client timeout 2213 ms). Trace:
I20260805 22:25:06.024854 31444 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:06.023300 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:06.028477 31711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56260 (request call id 2327) took 1243 ms. Trace:
I20260805 22:25:05.987324 3208 raft_consensus.cc:2726] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:06.029875 31704 rpcz_store.cc:276] 0805 22:25:04.288043 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:04.288546 (+ 503us) service_pool.cc:227] Handling call
0805 22:25:04.299766 (+ 11220us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:05.423778 (+1124012us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:05.977786 (+554008us) spinlock_profiling.cc:243] Waited 1.12 s on lock 0x7b54002b25d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:05.977957 (+ 171us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2583276800}
I20260805 22:25:06.050940 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:05.915722 3306 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:05.888096 3119 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:06.016189 3321 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:06.038405 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:06.061692 31603 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:06.039577 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.061812 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:06.062433 31502 raft_consensus.cc:2379] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 7: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:06.040829 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.030586 3235 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:06.045792 31594 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:25:06.048842 32591 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:05.988636 31714 rpcz_store.cc:276] 0805 22:25:04.786460 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.986887 (+1200427us) service_pool.cc:227] Handling call
0805 22:25:05.988288 (+ 1401us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:06.001101 31667 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:05.958006 3308 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:06.053606 31711 rpcz_store.cc:276] 0805 22:25:04.784748 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.027438 (+1242690us) service_pool.cc:227] Handling call
0805 22:25:06.028423 (+ 985us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:06.071458 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:06.071844 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:25:06.053993 31709 rpcz_store.cc:269] 0805 22:25:03.968276 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:03.968388 (+ 112us) service_pool.cc:227] Handling call
0805 22:25:06.049673 (+2081285us) spinlock_profiling.cc:243] Waited 88.2 ms on lock 0x7b54000ed154. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:06.049820 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":202790272}
I20260805 22:25:06.073257 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:06.058408 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:06.020570 2762 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:06.075428 31714 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:25:06.022198 3196 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:06.082018 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:06.082799 31594 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 7.
I20260805 22:25:06.083738 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:06.084376 31594 raft_consensus.cc:2454] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:25:06.066951 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:25:06.075503 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 74554.5us
I20260805 22:25:06.087277 31667 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:06.088523 3121 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:06.059165 32529 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:06.089419 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:06.074491 31715 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:25:06.078076 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:06.089457 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:06.076064 31714 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56260 (request call id 2334) took 1054 ms. Trace:
I20260805 22:25:06.096246 31714 rpcz_store.cc:276] 0805 22:25:05.021302 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.071194 (+1049892us) service_pool.cc:227] Handling call
0805 22:25:06.076004 (+ 4810us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:06.098390 32529 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:06.099486 31667 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:06.094393 31711 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:25:06.058084 3312 raft_consensus.cc:2781] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:06.103346 31709 raft_consensus.cc:3037] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:06.104132 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.020814 3309 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:06.044744 3008 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:06.032780 3131 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:06.111742 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.106899 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:06.074319 3321 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:06.108672 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:25:06.078926 3268 consensus_meta.cc:287] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.569s user 0.017s sys 0.003s
I20260805 22:25:06.116920 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:06.118713 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:06.101450 31601 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
W20260805 22:25:06.109994 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 127485us
I20260805 22:25:06.113713 31709 raft_consensus.cc:2454] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:25:06.114677 31603 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 9.
I20260805 22:25:06.121001 31444 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:06.048347 3221 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:06.055768 3272 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:06.122635 31554 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:06.123663 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:06.124249 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:06.115679 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:06.098824 3121 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:06.099527 3120 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:06.117233 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:06.103366 3173 raft_consensus.cc:2726] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:25:06.046271 3317 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:06.107785 3312 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
W20260805 22:25:06.155030 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 55610.2us
I20260805 22:25:06.082944 3235 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:06.137364 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:06.135254 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:06.155336 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:06.154788 31714 raft_consensus.cc:2454] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:25:06.115527 3144 raft_consensus.cc:2781] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:06.165720 31715 raft_consensus.cc:2379] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 7: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:06.157761 31667 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:06.158011 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.158254 31604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1324) took 1136 ms. Trace:
I20260805 22:25:06.166476 31604 rpcz_store.cc:276] 0805 22:25:05.021698 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.125201 (+1103503us) service_pool.cc:227] Handling call
0805 22:25:06.157774 (+ 32573us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:06.157950 (+ 176us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:06.157974 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:06.158013 (+ 39us) raft_consensus.cc:1631] Marking committed up to 7
0805 22:25:06.158045 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:06.158105 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:06.158200 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:06.167618 31713 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 8.
I20260805 22:25:06.120357 3321 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:06.160637 32579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57366 (request call id 556) took 1895 ms. Trace:
I20260805 22:25:06.180323 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:06.151599 2628 raft_consensus.cc:2781] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:25:06.152737 3120 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:06.162858 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:06.160578 31601 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
I20260805 22:25:06.164429 31714 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56260 (request call id 2330) took 1378 ms. Trace:
I20260805 22:25:06.169680 3010 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:06.170079 3052 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:06.170670 3098 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader election won for term 10
I20260805 22:25:06.171968 2643 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:06.172288 3306 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:06.128306 3322 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election won for term 8
W20260805 22:25:06.173183 3142 consensus_meta.cc:287] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 1.263s user 0.016s sys 0.007s
I20260805 22:25:06.174535 3235 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:06.174919 3309 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:06.175981 3317 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:06.176446 2762 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:06.160127 3312 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:06.177933 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.178290 31709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1943) took 1358 ms. Trace:
W20260805 22:25:06.178964 3325 log.cc:924] Time spent T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0: Append to log took a long time: real 0.139s user 0.001s sys 0.001s
W20260805 22:25:06.179124 31597 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59028 (request call id 2356) took 1845 ms (client timeout 2156 ms). Trace:
I20260805 22:25:06.180285 32584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57366 (request call id 550) took 2399 ms. Trace:
I20260805 22:25:06.180611 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:06.181951 3229 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:06.182363 31708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 2319) took 2214 ms. Trace:
I20260805 22:25:06.174474 3221 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:06.184841 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.185031 31494 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:06.387725 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.386790 32587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57366 (request call id 549) took 2606 ms. Trace:
I20260805 22:25:06.388742 32587 rpcz_store.cc:276] 0805 22:25:03.780139 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:03.790188 (+ 10049us) service_pool.cc:227] Handling call
0805 22:25:03.790539 (+ 351us) raft_consensus.cc:1457] Updating replica for 2 ops
0805 22:25:03.790951 (+ 412us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:03.790978 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0805 22:25:03.791572 (+ 594us) log.cc:842] Serialized 52 byte log entry
0805 22:25:04.021825 (+230253us) raft_consensus.cc:1631] Marking committed up to 7
0805 22:25:04.105684 (+ 83859us) log.cc:842] Serialized 14 byte log entry
0805 22:25:04.109546 (+ 3862us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:04.109640 (+ 94us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:05.976227 (+1866587us) spinlock_profiling.cc:243] Waited 2.45 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:25:06.184525 (+208298us) raft_consensus.cc:1671] finished
0805 22:25:06.217652 (+ 33127us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:06.386716 (+169064us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35296,"prepare.queue_time_us":101611,"prepare.run_cpu_time_us":40,"prepare.run_wall_time_us":42,"raft.queue_time_us":6517,"raft.run_cpu_time_us":200,"raft.run_wall_time_us":26469,"spinlock_wait_cycles":18140160,"thread_start_us":202413,"threads_started":2,"wal-append.queue_time_us":293620}
I20260805 22:25:06.387056 32584 rpcz_store.cc:276] 0805 22:25:03.780386 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:03.818254 (+ 37868us) service_pool.cc:227] Handling call
0805 22:25:06.179605 (+2361351us) spinlock_profiling.cc:243] Waited 9.4 ms on lock 0x7b5800005198. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:25:06.179819 (+ 214us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:06.179995 (+ 176us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:06.180018 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:06.180056 (+ 38us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:25:06.180082 (+ 26us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:06.180134 (+ 52us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:06.180234 (+ 100us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":21617536}
W20260805 22:25:06.386584 31597 rpcz_store.cc:269] 0805 22:25:04.333908 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.459338 (+1125430us) service_pool.cc:227] Handling call
0805 22:25:06.178940 (+719602us) spinlock_profiling.cc:243] Waited 161 ms on lock 0x7b54001437d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:06.179067 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":371142016}
I20260805 22:25:06.387143 2628 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:06.395496 2628 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:06.395622 31708 rpcz_store.cc:276] 0805 22:25:03.967995 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:03.968752 (+ 757us) service_pool.cc:227] Handling call
0805 22:25:03.985995 (+ 17243us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:04.900314 (+914319us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:04.900347 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:04.900390 (+ 43us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:25:04.900421 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:06.182117 (+1281696us) spinlock_profiling.cc:243] Waited 914 ms on lock 0x7b5400314f54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:06.182160 (+ 43us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:06.182270 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2102191360}
I20260805 22:25:06.397076 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:06.397711 31708 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:06.398053 3008 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:06.122694 3268 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:06.402508 32591 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:25:06.403395 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:06.403919 31444 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:06.404119 31494 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 10.
I20260805 22:25:06.404103 31597 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 10.
I20260805 22:25:06.403844 31709 rpcz_store.cc:276] 0805 22:25:04.819554 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.072994 (+1253440us) service_pool.cc:227] Handling call
0805 22:25:06.178130 (+105136us) spinlock_profiling.cc:243] Waited 29.5 ms on lock 0x7b5400032fd4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:06.178233 (+ 103us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":67758336}
I20260805 22:25:06.404630 31499 raft_consensus.cc:2396] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 8 because replica has last-logged OpId of term: 5 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 3.
I20260805 22:25:06.154325 3121 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:06.405494 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:06.406082 31444 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:06.406745 31597 raft_consensus.cc:2379] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 10: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
W20260805 22:25:06.269188 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 163023us
I20260805 22:25:06.184885 32579 rpcz_store.cc:276] 0805 22:25:04.265722 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:04.266425 (+ 703us) service_pool.cc:227] Handling call
0805 22:25:06.160477 (+1894052us) spinlock_profiling.cc:243] Waited 76.1 ms on lock 0x7b54004a06d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:06.160581 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":175078656}
I20260805 22:25:06.336946 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.411134 31604 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260805 22:25:06.402261 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.190120 31495 raft_consensus.cc:3037] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 8
I20260805 22:25:06.266330 31667 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:06.265079 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:06.407488 31714 rpcz_store.cc:276] 0805 22:25:04.785658 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.123046 (+1337388us) service_pool.cc:227] Handling call
0805 22:25:06.164359 (+ 41313us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:06.272300 31608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 2334) took 4240 ms. Trace:
W20260805 22:25:06.181974 3198 log.cc:924] Time spent T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 3.004s user 0.010s sys 0.017s
I20260805 22:25:06.415741 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:06.451581 31709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1942) took 1632 ms. Trace:
I20260805 22:25:06.451972 31709 rpcz_store.cc:276] 0805 22:25:04.819249 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.415492 (+1596243us) service_pool.cc:227] Handling call
0805 22:25:06.451507 (+ 36015us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260805 22:25:06.336095 3331 log.cc:924] Time spent T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.317s user 0.001s sys 0.000s
W20260805 22:25:06.416482 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 150220us
I20260805 22:25:06.165614 3144 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:06.420750 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.416626 31715 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:25:06.418790 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.416824 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:06.435369 31495 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:25:06.438050 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:25:06.267558 3333 log.cc:924] Time spent T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.211s user 0.001s sys 0.000s
I20260805 22:25:06.453116 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:06.414412 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:06.404584 3120 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:06.554312 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:25:06.220914 3298 consensus_meta.cc:287] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.626s user 0.011s sys 0.001s
I20260805 22:25:06.533641 31668 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:06.590660 3298 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:06.535066 31667 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:06.419780 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:06.540707 31708 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:25:06.265393 3173 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:06.221095 3207 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:06.416330 3142 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:06.458361 2762 raft_consensus.cc:2726] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:06.426867 3098 raft_consensus.cc:686] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:06.190512 3308 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:06.554960 31709 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:25:06.590026 32579 raft_consensus.cc:2354] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5.
I20260805 22:25:06.332036 3203 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:06.419812 3309 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260805 22:25:06.415114 3126 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:06.436045 3229 raft_consensus.cc:2781] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260805 22:25:06.416762 31608 rpcz_store.cc:276] 0805 22:25:01.982556 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:01.982822 (+ 266us) service_pool.cc:227] Handling call
0805 22:25:02.000494 (+ 17672us) raft_consensus.cc:1457] Updating replica for 3 ops
0805 22:25:02.002585 (+ 2091us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:02.002615 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 3 ops
0805 22:25:02.031026 (+ 28411us) log.cc:842] Serialized 78 byte log entry
0805 22:25:02.462874 (+431848us) raft_consensus.cc:1631] Marking committed up to 8
0805 22:25:02.547687 (+ 84813us) log.cc:842] Serialized 14 byte log entry
0805 22:25:03.669332 (+1121645us) log.cc:842] Serialized 14 byte log entry
0805 22:25:03.669642 (+ 310us) log.cc:842] Serialized 14 byte log entry
0805 22:25:06.183367 (+2513725us) spinlock_profiling.cc:243] Waited 878 us on lock 0x7b18002317c0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e10df96 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2730ffbfb3 00007f2731068f0f 00007f273106799e
0805 22:25:06.217678 (+ 34311us) spinlock_profiling.cc:243] Waited 6.3 ms on lock 0x7b54001c73a0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2730ffbfb3 00007f2731068f0f 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c
0805 22:25:06.217751 (+ 73us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:06.217852 (+ 101us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:06.217897 (+ 45us) raft_consensus.cc:1671] finished
0805 22:25:06.217908 (+ 11us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:06.222638 (+ 4730us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":69312,"prepare.queue_time_us":108595,"prepare.run_cpu_time_us":114,"prepare.run_wall_time_us":117,"raft.queue_time_us":244377,"raft.run_cpu_time_us":638,"raft.run_wall_time_us":82422,"spinlock_wait_cycles":16519296,"thread_start_us":1509329,"threads_started":2,"wal-append.queue_time_us":701441,"wal-append.run_cpu_time_us":21343,"wal-append.run_wall_time_us":2102263}
I20260805 22:25:06.557552 3144 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:06.556991 3303 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:06.597812 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.593710 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:06.713591 31714 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1938) took 2325 ms. Trace:
I20260805 22:25:06.713946 31714 rpcz_store.cc:276] 0805 22:25:04.388208 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.712771 (+2324563us) service_pool.cc:227] Handling call
0805 22:25:06.713105 (+ 334us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:06.713277 (+ 172us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:06.713301 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:06.713340 (+ 39us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:25:06.713368 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:06.713437 (+ 69us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:06.713531 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:06.715662 31714 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1939) took 2327 ms. Trace:
I20260805 22:25:06.715987 31714 rpcz_store.cc:276] 0805 22:25:04.388454 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.714942 (+2326488us) service_pool.cc:227] Handling call
0805 22:25:06.715232 (+ 290us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:06.715386 (+ 154us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:06.715410 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:06.715449 (+ 39us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:25:06.715477 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:06.715522 (+ 45us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:06.715611 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:06.605841 32589 raft_consensus.cc:1226] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5. Ops: []
I20260805 22:25:06.674007 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.674425 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:06.750661 31715 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:06.613620 31596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 2326) took 4669 ms. Trace:
W20260805 22:25:06.757018 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 82119.1us
I20260805 22:25:06.757763 31608 raft_consensus.cc:1261] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260805 22:25:06.678750 31601 raft_consensus.cc:1261] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260805 22:25:06.686184 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.593688 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:06.655030 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:06.790499 31501 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:06.657536 31493 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 7 index: 9. (index mismatch)
I20260805 22:25:06.752903 31596 rpcz_store.cc:276] 0805 22:25:01.943890 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:01.945474 (+ 1584us) service_pool.cc:227] Handling call
0805 22:25:01.945787 (+ 313us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:01.976685 (+ 30898us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:01.976715 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:02.375510 (+398795us) spinlock_profiling.cc:243] Waited 2.08 ms on lock 0x7b2c00000238. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272fa4be11 00007f272fa5b442 00007f27310a3c11 00007f273106858a 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772
0805 22:25:02.384822 (+ 9312us) log.cc:842] Serialized 26 byte log entry
0805 22:25:02.846096 (+461274us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:25:02.895498 (+ 49402us) log.cc:842] Serialized 14 byte log entry
0805 22:25:02.896888 (+ 1390us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:02.896965 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:06.089521 (+3192556us) spinlock_profiling.cc:243] Waited 41.3 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:25:06.613311 (+523790us) raft_consensus.cc:1671] finished
0805 22:25:06.613341 (+ 30us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:06.613546 (+ 205us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":49141,"prepare.run_cpu_time_us":97,"prepare.run_wall_time_us":2981,"raft.queue_time_us":38888,"raft.run_cpu_time_us":2001,"raft.run_wall_time_us":107438,"spinlock_wait_cycles":99798784,"thread_start_us":430549,"threads_started":1,"wal-append.queue_time_us":732516}
W20260805 22:25:06.865760 31603 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.673s user 0.065s sys 0.126s
I20260805 22:25:06.884868 31609 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1322) took 2066 ms. Trace:
I20260805 22:25:06.809729 31714 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1940) took 2421 ms. Trace:
I20260805 22:25:06.593354 3298 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:06.593962 3173 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:06.919042 31603 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.726s user 0.073s sys 0.130s
I20260805 22:25:06.830093 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.594005 3322 raft_consensus.cc:686] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 8 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:06.673707 3343 raft_consensus.cc:2781] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election won for term 8
I20260805 22:25:06.921926 3322 consensus_queue.cc:237] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:06.924299 3298 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:06.827026 31708 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:06.919432 31714 rpcz_store.cc:276] 0805 22:25:04.388701 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.716765 (+2328064us) service_pool.cc:227] Handling call
0805 22:25:06.717043 (+ 278us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:06.717221 (+ 178us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:06.717244 (+ 23us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:06.717281 (+ 37us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:25:06.717313 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:06.717357 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:06.809651 (+ 92294us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:06.832720 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.930806 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:06.843271 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.614738 3272 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:06.605599 3142 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:06.648370 3229 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:06.934674 3229 raft_consensus.cc:3037] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:25:06.917933 31609 rpcz_store.cc:276] 0805 22:25:04.818356 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.856223 (+1037867us) service_pool.cc:227] Handling call
0805 22:25:05.962317 (+106094us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:05.962525 (+ 208us) raft_consensus.cc:1520] Early marking committed up to index 8
0805 22:25:06.063624 (+101099us) log.cc:842] Serialized 14 byte log entry
0805 22:25:06.126314 (+ 62690us) log.cc:842] Serialized 14 byte log entry
0805 22:25:06.400301 (+273987us) spinlock_profiling.cc:243] Waited 1.66 ms on lock 0x7b180006a360. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e0b74ca 00007f2737798f99 00007f272df7b990 00007f272e13d864 00007f272e13d155 00007f27310603d0 00007f2731072576 00007f27310830d7 00007f2731083058
0805 22:25:06.884462 (+484161us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:06.884527 (+ 65us) raft_consensus.cc:1631] Marking committed up to 8
0805 22:25:06.884576 (+ 49us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:06.884653 (+ 77us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:06.884799 (+ 146us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":27911,"prepare.queue_time_us":188311,"prepare.run_cpu_time_us":35,"prepare.run_wall_time_us":36,"raft.queue_time_us":281764,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":16734,"spinlock_wait_cycles":24151936,"thread_start_us":601055,"threads_started":3,"wal-append.queue_time_us":200721}
I20260805 22:25:06.938686 3197 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:06.939266 3197 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:06.626484 3119 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:06.941306 3197 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:06.940696 31714 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1944) took 2072 ms. Trace:
I20260805 22:25:06.637945 3309 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:06.631242 3268 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:25:06.856212 3205 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:06.645859 3126 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260805 22:25:06.611585 3342 log.cc:924] Time spent T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.231s user 0.000s sys 0.000s
I20260805 22:25:06.825011 3312 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:06.948588 3312 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:06.805872 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 48051us
I20260805 22:25:06.943982 31714 rpcz_store.cc:276] 0805 22:25:04.867407 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.931814 (+2064407us) service_pool.cc:227] Handling call
0805 22:25:06.935218 (+ 3404us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:06.937133 (+ 1915us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:06.937338 (+ 205us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:06.937782 (+ 444us) raft_consensus.cc:1631] Marking committed up to 3
0805 22:25:06.938239 (+ 457us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:06.938792 (+ 553us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:06.940041 (+ 1249us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:06.957248 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:06.610807 3098 consensus_queue.cc:237] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:06.646400 3196 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:06.925032 3298 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:06.967823 3298 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:06.968482 3205 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:06.944070 3309 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:06.945430 3126 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:06.970105 31499 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:25:06.625981 3203 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:06.978631 3173 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:06.984009 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:06.985122 31608 raft_consensus.cc:1261] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 8 index: 6. (index mismatch)
W20260805 22:25:07.019502 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 45cd56d4e96040f98dbf1ae001a1005a, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:07.032997 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 45cd56d4e96040f98dbf1ae001a1005a, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:06.872804 3321 consensus_meta.cc:287] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.575s user 0.008s sys 0.000s
I20260805 22:25:07.017117 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:06.993306 31715 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 8.
W20260805 22:25:07.007383 31444 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.600s user 0.008s sys 0.028s
W20260805 22:25:06.807186 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 684562us
I20260805 22:25:07.040730 31715 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1950) took 1080 ms. Trace:
W20260805 22:25:07.040791 31444 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.634s user 0.008s sys 0.032s
W20260805 22:25:07.041354 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 635295us
I20260805 22:25:07.040280 32579 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
I20260805 22:25:07.028908 3205 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:06.601892 3207 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:07.018733 31594 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.618s user 0.005s sys 0.004s
I20260805 22:25:07.041074 31715 rpcz_store.cc:276] 0805 22:25:05.960069 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.543177 (+583108us) service_pool.cc:227] Handling call
0805 22:25:07.040672 (+497495us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260805 22:25:07.048708 31594 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.648s user 0.005s sys 0.004s
I20260805 22:25:06.987134 3343 raft_consensus.cc:686] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:07.036471 31608 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 2373) took 1011 ms. Trace:
I20260805 22:25:07.051504 31608 rpcz_store.cc:276] 0805 22:25:06.024973 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.758714 (+733741us) service_pool.cc:227] Handling call
0805 22:25:06.984803 (+226089us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:07.032872 (+ 48069us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:07.036413 (+ 3541us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:07.052543 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:07.044616 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:07.055037 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:25:07.045166 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 510151us
I20260805 22:25:07.056597 31667 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:07.041999 31554 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:25:07.010128 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 476563us
I20260805 22:25:07.057840 31668 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:07.058343 32529 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:07.059782 32529 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:25:07.064357 3121 consensus_meta.cc:287] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.531s user 0.008s sys 0.000s
W20260805 22:25:07.069236 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 49bb7cd9b51b4dd782a68b560d7570f2, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:07.069650 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 49bb7cd9b51b4dd782a68b560d7570f2, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:07.071521 31669 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.544s user 0.015s sys 0.117s
W20260805 22:25:07.072070 31669 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.548s user 0.016s sys 0.117s
I20260805 22:25:07.049624 31715 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58348 (request call id 600) took 2027 ms. Trace:
I20260805 22:25:07.069293 31554 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:07.072692 31715 rpcz_store.cc:276] 0805 22:25:05.022023 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:07.048876 (+2026853us) service_pool.cc:227] Handling call
0805 22:25:07.049176 (+ 300us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:07.049325 (+ 149us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:07.049345 (+ 20us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:07.049399 (+ 54us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:25:07.049428 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:07.049477 (+ 49us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:07.049571 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:07.049518 3312 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:07.076671 31554 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:07.077837 31554 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:07.068274 3142 raft_consensus.cc:2726] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:07.066766 3173 raft_consensus.cc:2726] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:25:07.109396 31668 leader_election.cc:398] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Vote denied by peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801) with higher term. Message: Invalid argument: T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5.
I20260805 22:25:07.114807 31668 leader_election.cc:401] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260805 22:25:07.112052 31708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 2328) took 2327 ms. Trace:
I20260805 22:25:07.116735 31715 raft_consensus.cc:1261] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 7 index: 7. (index mismatch)
I20260805 22:25:07.125950 31715 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58348 (request call id 603) took 1643 ms. Trace:
I20260805 22:25:07.126416 31715 rpcz_store.cc:276] 0805 22:25:05.482676 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:07.074032 (+1591356us) service_pool.cc:227] Handling call
0805 22:25:07.113717 (+ 39685us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:07.125699 (+ 11982us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:07.125895 (+ 196us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:07.114423 31603 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56384 (request call id 407) took 1276 ms. Trace:
I20260805 22:25:07.071929 3351 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election won for term 9
I20260805 22:25:07.076714 2948 raft_consensus.cc:2781] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:25:07.048167 3321 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.121578 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:07.071823 3356 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:07.049926 3120 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.122988 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:07.123015 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:07.137897 31493 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:07.089586 3047 raft_consensus.cc:2781] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:07.077306 3050 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:07.070214 3121 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.127460 31603 rpcz_store.cc:276] 0805 22:25:05.838492 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.164096 (+325604us) service_pool.cc:227] Handling call
0805 22:25:06.191592 (+ 27496us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:06.192253 (+ 661us) raft_consensus.cc:1520] Early marking committed up to index 9
0805 22:25:06.920177 (+727924us) log.cc:842] Serialized 14 byte log entry
0805 22:25:07.114049 (+193872us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:07.114109 (+ 60us) raft_consensus.cc:1631] Marking committed up to 9
0805 22:25:07.114156 (+ 47us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:07.114230 (+ 74us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:07.114355 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":873695,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":40,"thread_start_us":907505,"threads_started":2}
I20260805 22:25:07.072330 3272 raft_consensus.cc:2781] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:25:07.130908 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:07.138511 32587 raft_consensus.cc:2379] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 4: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:07.123286 3357 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:07.139349 32586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57372 (request call id 687) took 4585 ms. Trace:
W20260805 22:25:07.139863 31500 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40728 (request call id 1881) took 4227 ms (client timeout 3188 ms). Trace:
I20260805 22:25:07.090756 2731 consensus_queue.cc:1035] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.041s
I20260805 22:25:07.094089 3268 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:07.144363 31711 raft_consensus.cc:2379] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 8: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:07.179117 31493 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 8.
I20260805 22:25:07.147751 31704 raft_consensus.cc:2354] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 8. Current term is 9.
I20260805 22:25:07.152376 31600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1314) took 2443 ms. Trace:
I20260805 22:25:07.183745 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:07.172084 31501 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:25:07.150416 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:07.154243 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:07.184345 31600 rpcz_store.cc:276] 0805 22:25:04.708691 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.842830 (+1134139us) service_pool.cc:227] Handling call
0805 22:25:05.888627 (+ 45797us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:06.044659 (+156032us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:06.044687 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:06.044727 (+ 40us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:25:06.044752 (+ 25us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:07.152137 (+1107385us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b54002756d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:07.152183 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:07.152316 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":358368000}
I20260805 22:25:07.185817 32594 raft_consensus.cc:2379] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
W20260805 22:25:07.166567 31500 rpcz_store.cc:269] 0805 22:25:02.906584 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:02.906738 (+ 154us) service_pool.cc:227] Handling call
0805 22:25:07.133635 (+4226897us) spinlock_profiling.cc:243] Waited 655 ms on lock 0x7b54000f7dd4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:07.133756 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1507304320}
I20260805 22:25:07.142385 31715 raft_consensus.cc:1261] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260805 22:25:07.186750 31715 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1949) took 1228 ms. Trace:
I20260805 22:25:07.144805 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:25:07.157169 31713 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1929) took 2771 ms (client timeout 2185 ms). Trace:
I20260805 22:25:07.187070 31715 rpcz_store.cc:276] 0805 22:25:05.958650 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:07.127281 (+1168631us) service_pool.cc:227] Handling call
0805 22:25:07.142053 (+ 14772us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:07.186525 (+ 44472us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:07.186700 (+ 175us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260805 22:25:07.187215 31713 rpcz_store.cc:269] 0805 22:25:04.385786 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.866168 (+1480382us) service_pool.cc:227] Handling call
0805 22:25:07.156982 (+1290814us) spinlock_profiling.cc:243] Waited 272 ms on lock 0x7b5400150c54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:07.157101 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":624814976}
W20260805 22:25:07.189548 32594 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57366 (request call id 561) took 1871 ms (client timeout 2458 ms). Trace:
I20260805 22:25:07.190124 32589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57382 (request call id 780) took 1230 ms. Trace:
W20260805 22:25:07.189929 32594 rpcz_store.cc:269] 0805 22:25:05.318400 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.319848 (+ 1448us) service_pool.cc:227] Handling call
0805 22:25:07.154087 (+1834239us) spinlock_profiling.cc:243] Waited 2.02 ms on lock 0x7b5800005198. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735faa1b0 00007f2735fa9a94 00007f272f691c04 00007f272f691b92
0805 22:25:07.189486 (+ 35399us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4641024}
I20260805 22:25:07.194095 31490 raft_consensus.cc:2379] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 9: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:07.090267 3312 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.184702 31493 raft_consensus.cc:2454] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
I20260805 22:25:07.166337 32586 rpcz_store.cc:276] 0805 22:25:02.553678 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:02.555839 (+ 2161us) service_pool.cc:227] Handling call
0805 22:25:02.583017 (+ 27178us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:02.674489 (+ 91472us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:02.674517 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:02.674902 (+ 385us) log.cc:842] Serialized 26 byte log entry
0805 22:25:03.525621 (+850719us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:25:03.525660 (+ 39us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:07.132903 (+3607243us) spinlock_profiling.cc:243] Waited 87 ms on lock 0x7b54004d1cd4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:07.132949 (+ 46us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:07.132996 (+ 47us) raft_consensus.cc:1671] finished
0805 22:25:07.133018 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:07.139263 (+ 6245us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3989,"raft.queue_time_us":4675,"raft.run_cpu_time_us":153,"raft.run_wall_time_us":11809,"spinlock_wait_cycles":200178432,"thread_start_us":806452,"threads_started":1,"wal-append.queue_time_us":821853,"wal-append.run_cpu_time_us":2006,"wal-append.run_wall_time_us":1088964}
I20260805 22:25:07.190481 32589 rpcz_store.cc:276] 0805 22:25:05.959571 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.057010 (+ 97439us) service_pool.cc:227] Handling call
0805 22:25:06.057263 (+ 253us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:06.718653 (+661390us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:07.189884 (+471231us) spinlock_profiling.cc:243] Waited 548 ms on lock 0x7b540017dc54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:07.190067 (+ 183us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1261054336}
W20260805 22:25:07.159886 2628 consensus_meta.cc:287] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.764s user 0.008s sys 0.000s
I20260805 22:25:07.227691 31490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40740 (request call id 1356) took 1057 ms. Trace:
I20260805 22:25:07.228173 31490 rpcz_store.cc:276] 0805 22:25:06.170620 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.180398 (+ 9778us) service_pool.cc:227] Handling call
0805 22:25:07.227442 (+1047044us) spinlock_profiling.cc:243] Waited 806 ms on lock 0x7b5400121e54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:07.227595 (+ 153us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1854181120}
I20260805 22:25:07.213114 3079 consensus_queue.cc:1046] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260805 22:25:07.230836 3079 raft_consensus.cc:3032] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:25:07.231267 3079 raft_consensus.cc:729] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:07.232542 3131 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:07.189677 3197 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:07.139967 3356 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:07.157725 3192 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:07.233428 3356 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:07.166136 3357 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:07.233567 3192 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:07.235477 3357 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:07.235631 3192 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:07.165661 3203 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:07.182912 3351 raft_consensus.cc:686] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:07.237074 3351 consensus_queue.cc:237] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.151248 32585 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57366 (request call id 562) took 1416 ms. Trace:
I20260805 22:25:07.238288 32585 rpcz_store.cc:276] 0805 22:25:05.735126 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.773301 (+ 38175us) service_pool.cc:227] Handling call
0805 22:25:05.858927 (+ 85626us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:06.081367 (+222440us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:06.090779 (+ 9412us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:06.104890 (+ 14111us) raft_consensus.cc:1631] Marking committed up to 7
0805 22:25:06.107626 (+ 2736us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:07.150980 (+1043354us) spinlock_profiling.cc:243] Waited 205 ms on lock 0x7b540017cad4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:07.151035 (+ 55us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:07.151179 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":471135488}
I20260805 22:25:07.151062 3361 consensus_queue.cc:1035] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.069s
W20260805 22:25:07.224733 31556 connection.cc:232] RPC call timeout handler was delayed by -2.0605969429016113s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1516s for a response
I20260805 22:25:07.140841 3363 raft_consensus.cc:2781] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:25:07.147019 3050 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.259114 3050 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:07.161698 3199 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:07.128055 3142 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.064s
I20260805 22:25:07.151392 3272 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:07.156409 3207 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:07.227452 2628 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:07.232112 3079 consensus_queue.cc:260] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:07.237540 3321 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:07.170118 3343 consensus_queue.cc:237] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 6.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:07.151551 3202 raft_consensus.cc:2726] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801) with higher term. Message: Invalid argument: T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5.
I20260805 22:25:07.144088 3120 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:07.244148 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:07.284444 32594 raft_consensus.cc:2454] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:25:07.245249 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:07.147034 3047 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:07.127411 31708 rpcz_store.cc:276] 0805 22:25:04.785070 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.826543 (+2041473us) service_pool.cc:227] Handling call
0805 22:25:06.826911 (+ 368us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:07.111712 (+284801us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:07.111743 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:07.111782 (+ 39us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:25:07.111816 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:07.111875 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:07.111993 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":58717,"raft.queue_time_us":97846,"raft.run_cpu_time_us":108,"raft.run_wall_time_us":109}
I20260805 22:25:07.148813 3121 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:07.247720 31715 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 2336) took 1248 ms. Trace:
I20260805 22:25:07.300881 31715 rpcz_store.cc:276] 0805 22:25:05.998898 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:07.188314 (+1189416us) service_pool.cc:227] Handling call
0805 22:25:07.247242 (+ 58928us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:07.247409 (+ 167us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:07.247435 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:07.247471 (+ 36us) raft_consensus.cc:1631] Marking committed up to 1
0805 22:25:07.247500 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:07.247559 (+ 59us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:07.247664 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260805 22:25:07.251904 31556 connection.cc:232] RPC call timeout handler was delayed by -1.9636759948916733s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.15660000000000002s for a response
I20260805 22:25:07.285730 31447 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
W20260805 22:25:07.301996 31556 connection.cc:232] RPC call timeout handler was delayed by -1.8797403238713741s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1686s for a response
W20260805 22:25:07.284292 31559 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.870s user 0.000s sys 0.012s
W20260805 22:25:07.280349 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 236570us
W20260805 22:25:07.302517 31559 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.896s user 0.000s sys 0.012s
I20260805 22:25:07.285422 32586 raft_consensus.cc:2454] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 3.
W20260805 22:25:07.302310 31556 connection.cc:232] RPC call timeout handler was delayed by -1.8522724020294845s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.19830000000000003s for a response
W20260805 22:25:07.304426 31556 connection.cc:232] RPC call timeout handler was delayed by -1.7266716449521482s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1845s for a response
I20260805 22:25:07.304728 31447 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:07.304981 31556 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:07.306586 31556 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:07.307710 31556 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:07.308708 31556 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:07.310400 3050 raft_consensus.cc:2781] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:25:07.310911 3050 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:07.311327 3050 raft_consensus.cc:3037] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:07.312435 3207 raft_consensus.cc:2781] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:25:07.312835 3207 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:07.313180 3207 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:25:07.314211 31602 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59030 (request call id 1310) took 3324 ms (client timeout 1914 ms). Trace:
I20260805 22:25:07.314525 3126 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.315927 3036 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:07.316465 3036 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:07.316607 3126 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:07.317061 3356 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.318575 3079 raft_consensus.cc:3037] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:07.319319 31604 raft_consensus.cc:1226] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 7. Current term is 8. Ops: []
I20260805 22:25:07.280587 3272 raft_consensus.cc:3037] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:25:07.314571 31602 rpcz_store.cc:269] 0805 22:25:03.989750 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:04.034784 (+ 45034us) service_pool.cc:227] Handling call
0805 22:25:07.313984 (+3279200us) spinlock_profiling.cc:243] Waited 94.6 ms on lock 0x7b540005d2d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:07.314142 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":217585408}
I20260805 22:25:07.317068 3036 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:07.208552 3268 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.319353 3356 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:07.233166 3197 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:07.298449 3363 raft_consensus.cc:686] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:07.250670 3229 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:07.355238 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 527d60ab6db74595bed7e87ca2ecc6e8, TS 516cb07ce3064ccbbca35e912d62816c -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:07.360684 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 527d60ab6db74595bed7e87ca2ecc6e8, TS 516cb07ce3064ccbbca35e912d62816c -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.303363 3047 raft_consensus.cc:3037] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:07.321700 2731 raft_consensus.cc:2726] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:07.326383 3357 raft_consensus.cc:2781] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260805 22:25:07.320572 3042 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:07.377233 31596 raft_consensus.cc:1226] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:25:07.323800 31714 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1945) took 2427 ms. Trace:
I20260805 22:25:07.369474 31709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1936) took 2981 ms. Trace:
I20260805 22:25:07.348918 3036 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:07.485085 31597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 2364) took 1519 ms. Trace:
I20260805 22:25:07.378168 31714 rpcz_store.cc:276] 0805 22:25:04.896535 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.952364 (+2055829us) service_pool.cc:227] Handling call
0805 22:25:06.955252 (+ 2888us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:06.956771 (+ 1519us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:06.956909 (+ 138us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:06.960286 (+ 3377us) log.cc:842] Serialized 26 byte log entry
0805 22:25:07.110446 (+150160us) raft_consensus.cc:1631] Marking committed up to 2
0805 22:25:07.110477 (+ 31us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:07.110549 (+ 72us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:07.323583 (+213034us) raft_consensus.cc:1671] finished
0805 22:25:07.323607 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:07.323741 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"thread_start_us":147983,"threads_started":1,"wal-append.queue_time_us":349034}
W20260805 22:25:07.449041 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 61034us
W20260805 22:25:07.378751 2944 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 4.161s user 0.017s sys 0.044s
I20260805 22:25:07.391762 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:07.470021 2948 raft_consensus.cc:686] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [term 7 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:07.470146 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:07.515277 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:07.431200 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:07.379938 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 255934us
I20260805 22:25:07.517333 31603 raft_consensus.cc:2379] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 9: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:07.497771 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:07.392186 31709 rpcz_store.cc:276] 0805 22:25:04.387673 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.615430 (+2227757us) service_pool.cc:227] Handling call
0805 22:25:06.615812 (+ 382us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:06.616016 (+ 204us) raft_consensus.cc:1520] Early marking committed up to index 6
0805 22:25:07.008370 (+392354us) log.cc:842] Serialized 14 byte log entry
0805 22:25:07.369096 (+360726us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:07.369168 (+ 72us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:25:07.369214 (+ 46us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:07.369281 (+ 67us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:07.369412 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":502322,"prepare.run_cpu_time_us":53,"prepare.run_wall_time_us":1769,"raft.queue_time_us":452,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":10053,"thread_start_us":641248,"threads_started":2,"wal-append.queue_time_us":423438}
I20260805 22:25:07.526425 31601 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:07.516177 31597 rpcz_store.cc:276] 0805 22:25:05.965608 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.418284 (+452676us) service_pool.cc:227] Handling call
0805 22:25:06.418695 (+ 411us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:06.418898 (+ 203us) raft_consensus.cc:1520] Early marking committed up to index 5
0805 22:25:07.001782 (+582884us) log.cc:842] Serialized 14 byte log entry
0805 22:25:07.484702 (+482920us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:07.484759 (+ 57us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:25:07.484811 (+ 52us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:07.484886 (+ 75us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:07.485011 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8790,"prepare.queue_time_us":655332,"prepare.run_cpu_time_us":46,"prepare.run_wall_time_us":49,"raft.queue_time_us":10251,"thread_start_us":930240,"threads_started":2}
W20260805 22:25:07.380615 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 308562us
I20260805 22:25:07.549301 31492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1907) took 2657 ms. Trace:
I20260805 22:25:07.558933 32529 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:07.551186 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 448758us
I20260805 22:25:07.560495 32529 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:07.431370 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:07.530400 31602 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:25:07.565506 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:07.566624 31667 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:07.567088 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:07.567891 31715 raft_consensus.cc:3037] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
W20260805 22:25:07.516710 2944 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 4.325s user 0.021s sys 0.049s
I20260805 22:25:07.593231 32529 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:07.591282 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:07.591742 31554 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:07.597875 32529 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:07.589812 31667 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:07.392666 3036 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:07.376626 2643 consensus_queue.cc:1035] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.218s
I20260805 22:25:07.376816 3357 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:07.601451 31554 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:07.632057 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:07.633623 31667 leader_election.cc:398] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Vote denied by peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607) with higher term. Message: Invalid argument: T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 6. Current term is 7.
I20260805 22:25:07.634866 31667 leader_election.cc:401] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Cancelling election due to peer responding with higher term
I20260805 22:25:07.636878 31667 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:07.631678 31486 raft_consensus.cc:595] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:07.494603 3308 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:25:07.413549 3144 consensus_meta.cc:287] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.637s user 0.009s sys 0.001s
I20260805 22:25:07.526458 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:07.638360 31601 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 9.
I20260805 22:25:07.641125 31714 raft_consensus.cc:2379] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:07.551147 3312 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:07.550760 3281 raft_consensus.cc:2781] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:07.590780 3298 raft_consensus.cc:2726] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:07.655890 31667 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:07.412540 3253 consensus_queue.cc:1035] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.219s
I20260805 22:25:07.599731 3173 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:25:07.658776 31444 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:07.599920 2855 raft_consensus.cc:2781] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:07.380295 3042 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.640334 3357 raft_consensus.cc:3037] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Advancing to term 3
I20260805 22:25:07.631330 3345 raft_consensus.cc:2781] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:07.526707 3205 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.681181 31667 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:07.682749 2953 raft_consensus.cc:2726] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:07.651789 3144 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:07.686036 3253 raft_consensus.cc:2781] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:07.686501 31444 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
W20260805 22:25:07.690587 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 89228.5us
I20260805 22:25:07.695900 31715 raft_consensus.cc:2454] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:25:07.665761 3121 raft_consensus.cc:2781] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:07.648526 3359 raft_consensus.cc:2726] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election lost for term 6. Reason: Vote denied by peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607) with higher term. Message: Invalid argument: T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 6. Current term is 7.
I20260805 22:25:07.639789 3192 consensus_queue.cc:1035] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.039s
I20260805 22:25:07.671926 3203 raft_consensus.cc:2781] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:07.672350 2855 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:07.681425 2628 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:07.704723 31601 raft_consensus.cc:1261] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260805 22:25:07.498992 3363 consensus_queue.cc:237] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:07.708365 31708 raft_consensus.cc:1226] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 8. Current term is 9. Ops: []
I20260805 22:25:07.676862 3303 raft_consensus.cc:2781] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:25:07.669618 3378 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election won for term 8
W20260805 22:25:07.791112 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 194663us
I20260805 22:25:07.710829 3203 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
W20260805 22:25:07.704033 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 106217us
I20260805 22:25:07.677927 3268 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:07.778296 3042 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:07.698048 3253 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:07.691237 3345 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
W20260805 22:25:07.716742 31706 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1928) took 3325 ms (client timeout 1780 ms). Trace:
I20260805 22:25:07.651345 3308 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.744182 31600 raft_consensus.cc:1226] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 8. Current term is 9. Ops: [8.6-8.6]
W20260805 22:25:07.703714 3309 consensus_meta.cc:287] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.667s user 0.010s sys 0.000s
W20260805 22:25:07.811949 31711 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 1948) took 1854 ms (client timeout 1864 ms). Trace:
W20260805 22:25:07.812345 31711 rpcz_store.cc:269] 0805 22:25:05.957913 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.118435 (+160522us) service_pool.cc:227] Handling call
0805 22:25:07.811750 (+1693315us) spinlock_profiling.cc:243] Waited 1.02 s on lock 0x7b54002b3ed4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:07.811890 (+ 140us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2356880768}
I20260805 22:25:07.813776 31713 raft_consensus.cc:1226] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 7. Current term is 8. Ops: []
I20260805 22:25:07.815394 31713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 2337) took 1816 ms. Trace:
W20260805 22:25:07.799562 31706 rpcz_store.cc:269] 0805 22:25:04.385414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.422466 (+1037052us) service_pool.cc:227] Handling call
0805 22:25:07.710382 (+2287916us) spinlock_profiling.cc:243] Waited 22.2 ms on lock 0x7b5400314054. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:07.710540 (+ 158us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":51133184}
I20260805 22:25:07.781816 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:07.676695 3356 consensus_queue.cc:1035] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.040s
I20260805 22:25:07.816910 3309 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.818691 3126 raft_consensus.cc:979] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994: : Instructing follower 516cb07ce3064ccbbca35e912d62816c to start an election
I20260805 22:25:07.819150 3126 raft_consensus.cc:1067] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Signalling peer 516cb07ce3064ccbbca35e912d62816c to start an election
I20260805 22:25:07.821116 3309 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 11 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:07.778084 3207 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:07.782240 3144 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:07.772816 3374 log.cc:924] Time spent T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.109s user 0.000s sys 0.000s
I20260805 22:25:07.823037 31705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 1923) took 3500 ms. Trace:
W20260805 22:25:07.777205 3379 log.cc:924] Time spent T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.212s user 0.001s sys 0.000s
I20260805 22:25:07.823383 31705 rpcz_store.cc:276] 0805 22:25:04.322572 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:04.322707 (+ 135us) service_pool.cc:227] Handling call
0805 22:25:05.801975 (+1479268us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:05.802193 (+ 218us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:05.802226 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:05.802268 (+ 42us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:25:05.802295 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:05.802348 (+ 53us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:07.822880 (+2020532us) spinlock_profiling.cc:243] Waited 1.46 s on lock 0x7b540016c950. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731067a09 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:07.822984 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3367466880}
I20260805 22:25:07.823961 3207 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:07.824146 31594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1327) took 2387 ms. Trace:
W20260805 22:25:07.716089 3369 log.cc:924] Time spent T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.303s user 0.000s sys 0.001s
I20260805 22:25:07.716272 3376 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:07.708570 2956 raft_consensus.cc:2781] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:25:07.702870 3121 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:07.705588 3201 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader election won for term 9
I20260805 22:25:07.825124 3376 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:07.825676 3121 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:07.825876 3201 raft_consensus.cc:686] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 9 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:07.827338 3376 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:07.826942 3201 consensus_queue.cc:237] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.824463 31594 rpcz_store.cc:276] 0805 22:25:05.437082 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.163096 (+726014us) service_pool.cc:227] Handling call
0805 22:25:06.163459 (+ 363us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:07.049096 (+885637us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:07.049126 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:07.056220 (+ 7094us) log.cc:842] Serialized 26 byte log entry
0805 22:25:07.366951 (+310731us) raft_consensus.cc:1631] Marking committed up to 7
0805 22:25:07.366989 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:07.367066 (+ 77us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:07.823938 (+456872us) raft_consensus.cc:1671] finished
0805 22:25:07.823959 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:07.824093 (+ 134us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":237165,"raft.queue_time_us":251541,"spinlock_wait_cycles":192739200,"thread_start_us":958404,"threads_started":2,"wal-append.queue_time_us":509299}
I20260805 22:25:07.558512 31492 rpcz_store.cc:276] 0805 22:25:04.892159 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:05.469998 (+577839us) service_pool.cc:227] Handling call
0805 22:25:05.480796 (+ 10798us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:05.480993 (+ 197us) raft_consensus.cc:1520] Early marking committed up to index 4
0805 22:25:07.146370 (+1665377us) spinlock_profiling.cc:243] Waited 9.73 ms on lock 0x7b18002fd280. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f2730cf782c 00007f272e1229dc 00007f2736f59a39 00007f272e13f8bb 00007f272e13dcbf 00007f272e13d155 00007f2734a3d167 00007f2731072807 00007f2731081f0f
0805 22:25:07.146511 (+ 141us) log.cc:842] Serialized 14 byte log entry
0805 22:25:07.548885 (+402374us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:07.548937 (+ 52us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:25:07.548981 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:07.549112 (+ 131us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:07.549239 (+ 127us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":90404,"prepare.queue_time_us":363712,"prepare.run_cpu_time_us":43,"prepare.run_wall_time_us":45,"raft.queue_time_us":92561,"spinlock_wait_cycles":22373632,"thread_start_us":613143,"threads_started":2}
I20260805 22:25:07.716754 2855 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:07.790448 32579 raft_consensus.cc:1261] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260805 22:25:07.703490 31667 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:25:07.831866 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 128648us
W20260805 22:25:07.832324 31494 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40740 (request call id 1349) took 1781 ms (client timeout 1851 ms). Trace:
I20260805 22:25:07.832568 31667 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
W20260805 22:25:07.793653 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 99477.3us
I20260805 22:25:07.795959 32529 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:07.797140 3321 consensus_queue.cc:1035] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.004s
I20260805 22:25:07.836233 31667 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:07.803321 3233 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:07.803650 3303 raft_consensus.cc:686] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:07.804119 3036 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:07.784088 2643 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:07.807293 3253 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:07.794433 3203 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:07.857968 3288 consensus_queue.cc:1046] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.003s
I20260805 22:25:07.815758 31713 rpcz_store.cc:276] 0805 22:25:05.999158 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:07.188349 (+1189191us) service_pool.cc:227] Handling call
0805 22:25:07.813612 (+625263us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:07.814159 (+ 547us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:07.815216 (+ 1057us) spinlock_profiling.cc:243] Waited 590 ms on lock 0x7b54002b3ed0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731067a09 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:07.815337 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1357824256}
I20260805 22:25:07.849172 3376 raft_consensus.cc:2726] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:07.822259 3205 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:07.858088 31603 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e"
dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:59028
I20260805 22:25:07.687674 2731 raft_consensus.cc:2741] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260805 22:25:07.687424 3229 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:07.859618 31603 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:25:07.824728 3047 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.860003 31603 raft_consensus.cc:3037] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:07.827266 2956 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:07.861585 3047 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:07.861747 2956 raft_consensus.cc:3037] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
W20260805 22:25:07.844924 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 49050us
I20260805 22:25:07.706321 2948 consensus_queue.cc:237] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.865821 32529 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:07.866644 3036 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:07.856376 3351 raft_consensus.cc:2726] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:07.869460 2643 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:07.870752 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:07.848034 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:07.871769 32579 raft_consensus.cc:2379] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:07.810662 3050 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:07.850435 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 8abfbdc4afff482ebc81c4fda5d0812a, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:07.875638 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 8abfbdc4afff482ebc81c4fda5d0812a, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.876116 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:07.877978 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:07.849574 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
W20260805 22:25:07.832728 31494 rpcz_store.cc:269] 0805 22:25:06.050919 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.051343 (+ 424us) service_pool.cc:227] Handling call
0805 22:25:07.825306 (+1773963us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x7b54000f73d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:07.832264 (+ 6958us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":303528832}
I20260805 22:25:07.825362 3308 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:07.872901 31444 leader_election.cc:334] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Timed out: RequestConsensusVote RPC to 127.29.21.195:44831 timed out after 1.658s (SENT)
I20260805 22:25:07.879675 31444 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:07.879691 31601 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:07.878607 31597 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:25:07.797231 3378 raft_consensus.cc:686] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 8 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:07.878549 31554 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:07.882577 31554 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:07.854372 31667 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:07.883879 31554 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:07.884789 3376 raft_consensus.cc:2781] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:25:07.885241 3376 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:07.885484 3288 raft_consensus.cc:3032] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 8 LEADER]: Stepping down as leader of term 8
I20260805 22:25:07.884804 3409 raft_consensus.cc:2726] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260805 22:25:07.885591 3376 raft_consensus.cc:3037] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:07.850749 3131 consensus_queue.cc:1035] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.025s
I20260805 22:25:07.888545 3303 consensus_queue.cc:237] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:07.891314 2643 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:07.872923 31603 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:07.891958 31594 raft_consensus.cc:2379] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 6: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:07.894234 31603 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:07.894109 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:07.897579 31600 raft_consensus.cc:1226] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
W20260805 22:25:07.899940 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 45657.5us
I20260805 22:25:07.902486 31667 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:07.848654 2628 raft_consensus.cc:2726] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:07.904206 31554 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:07.844900 3359 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:07.845238 3345 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:07.917438 31667 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
W20260805 22:25:07.918218 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 49357.5us
W20260805 22:25:07.919142 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 46266.2us
I20260805 22:25:07.926637 2953 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260805 22:25:07.918242 3268 raft_consensus.cc:2741] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260805 22:25:07.927677 2643 raft_consensus.cc:2726] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:07.928236 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:07.928721 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:07.928965 32587 raft_consensus.cc:2379] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:07.930214 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:07.930884 32587 raft_consensus.cc:2379] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 4: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:07.931902 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:07.933751 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:07.869490 3357 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:07.927084 2953 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:07.935829 2953 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:07.936925 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:07.937536 32589 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:25:07.937726 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:07.938309 32591 raft_consensus.cc:2454] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
I20260805 22:25:07.938546 31444 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:07.941747 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:07.942883 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:07.943885 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:07.944067 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:07.944773 32586 raft_consensus.cc:2354] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5.
I20260805 22:25:07.945043 32583 raft_consensus.cc:1226] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:07.946293 3411 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:07.946969 3411 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:07.944918 32529 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:07.935290 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:07.950258 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:07.885936 3288 raft_consensus.cc:729] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:07.952834 3298 raft_consensus.cc:2726] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260805 22:25:07.963373 3377 log.cc:924] Time spent T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.274s user 0.001s sys 0.000s
I20260805 22:25:07.904995 3229 raft_consensus.cc:2726] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:07.954528 3050 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:07.958490 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:07.959395 31667 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:07.968703 3351 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:25:07.951740 32586 raft_consensus.cc:3037] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:07.971652 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:07.972297 31713 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
I20260805 22:25:07.972489 31554 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:07.812386 3272 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:07.973351 3229 raft_consensus.cc:2781] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:07.974124 3229 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:07.974749 3229 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:07.975968 3272 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:07.976882 31709 raft_consensus.cc:2379] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:07.922329 3317 raft_consensus.cc:2726] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:07.962687 3142 consensus_queue.cc:1035] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:25:07.963219 31707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58348 (request call id 591) took 4781 ms. Trace:
I20260805 22:25:07.979246 31707 rpcz_store.cc:276] 0805 22:25:03.181762 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:03.182533 (+ 771us) service_pool.cc:227] Handling call
0805 22:25:03.182883 (+ 350us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:03.227062 (+ 44179us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:03.227092 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:03.259232 (+ 32140us) log.cc:842] Serialized 26 byte log entry
0805 22:25:05.835841 (+2576609us) spinlock_profiling.cc:243] Waited 1.4 ms on lock 0x7b1800414ac0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e122345 00007f2736f59a39 00007f272e13f8bb 00007f272e13dcbf 00007f272e13fe22 00007f2730ce95bb 00007f2730cef063 00007f2730cef411 00007f273103c5cf
0805 22:25:05.986586 (+150745us) raft_consensus.cc:1631] Marking committed up to 8
0805 22:25:05.986627 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:07.962826 (+1976199us) spinlock_profiling.cc:243] Waited 26.8 ms on lock 0x7b5400203d54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:07.962882 (+ 56us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:07.962932 (+ 50us) raft_consensus.cc:1671] finished
0805 22:25:07.962940 (+ 8us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:07.963097 (+ 157us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9976,"raft.queue_time_us":19357,"raft.run_cpu_time_us":218,"raft.run_wall_time_us":66595,"spinlock_wait_cycles":64829056,"thread_start_us":150550,"threads_started":1,"wal-append.queue_time_us":2865767,"wal-append.run_cpu_time_us":2233,"wal-append.run_wall_time_us":1006156}
I20260805 22:25:07.982892 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:07.965090 32529 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:07.982892 32589 raft_consensus.cc:1261] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260805 22:25:07.983992 3142 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:07.977837 2953 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:07.985288 32529 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:25:07.986068 3142 raft_consensus.cc:2781] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:25:07.985704 3272 consensus_queue.cc:1035] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.001s
I20260805 22:25:07.919546 3361 raft_consensus.cc:3037] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:07.987177 3142 raft_consensus.cc:686] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:07.976926 31601 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 11.
I20260805 22:25:07.987648 2953 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:07.989099 3142 consensus_queue.cc:237] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 4.7, Last appended by leader: 7, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:07.991021 2731 consensus_queue.cc:1035] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.001s
I20260805 22:25:07.986366 32586 raft_consensus.cc:2454] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
W20260805 22:25:07.973289 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 47711.1us
I20260805 22:25:07.995806 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:25:07.996254 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 43859.1us
W20260805 22:25:07.997856 31444 leader_election.cc:334] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Timed out: RequestConsensusVote RPC to 127.29.21.195:44831 timed out after 1.774s (SENT)
I20260805 22:25:07.998279 31444 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:07.997974 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:07.999935 32529 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:07.994618 31667 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:08.001560 31667 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:08.002336 2953 raft_consensus.cc:2726] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:25:08.002578 3298 raft_consensus.cc:2781] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:08.008816 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:08.016475 31554 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:08.017988 31554 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:08.020753 31444 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:08.023311 31556 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:08.023306 31667 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:08.027441 2643 raft_consensus.cc:2726] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:25:08.028443 3375 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:08.029548 3308 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Leader election won for term 11
I20260805 22:25:08.003628 3357 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:08.049142 31667 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:08.016811 3298 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:07.996858 3376 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:08.025771 2953 raft_consensus.cc:2726] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:08.019682 2731 consensus_queue.cc:1035] T 45cd56d4e96040f98dbf1ae001a1005a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.027s
I20260805 22:25:08.046200 3272 raft_consensus.cc:2726] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:08.016666 3050 consensus_queue.cc:1046] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 1.1, Next index: 2, Last known committed idx: 1, Time since last communication: 0.042s
I20260805 22:25:08.057557 3208 raft_consensus.cc:2726] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:08.077487 32591 raft_consensus.cc:2379] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 4: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:08.060676 3298 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:08.082962 3411 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:08.083791 3197 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:08.045680 2956 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:08.086104 3345 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:08.086848 2956 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:08.088050 3345 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:08.056438 3378 consensus_queue.cc:237] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:08.062366 3308 raft_consensus.cc:686] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 11 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:08.083014 3376 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:08.087172 3121 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:08.050232 3288 consensus_queue.cc:260] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:08.146668 3121 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:08.147943 3229 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:08.151721 3229 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:08.169453 3203 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:08.176978 3419 log.cc:924] Time spent T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.061s user 0.000s sys 0.001s
I20260805 22:25:08.180289 3288 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:08.176162 3203 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:08.030839 31444 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:08.258517 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:08.262421 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
W20260805 22:25:08.304028 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 274520us
I20260805 22:25:08.263926 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:08.183759 3253 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:08.185398 3308 consensus_queue.cc:237] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:08.187834 3361 raft_consensus.cc:2726] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:25:08.274399 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:08.267930 32589 raft_consensus.cc:1226] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
I20260805 22:25:08.276955 3207 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:08.270682 2956 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:08.265241 3045 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:08.266646 2842 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:08.324338 3045 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:08.325253 3036 raft_consensus.cc:2726] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:25:08.325187 2842 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:08.327802 3045 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:08.090131 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:25:08.306454 3439 log.cc:924] Time spent T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.081s user 0.000s sys 0.001s
I20260805 22:25:08.329663 2956 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:08.306691 32585 raft_consensus.cc:2379] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:08.315769 3235 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:08.317252 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:08.318030 31706 raft_consensus.cc:1226] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 4. Current term is 5. Ops: []
I20260805 22:25:08.324982 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:08.333894 31494 raft_consensus.cc:2379] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 9: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:08.321600 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:08.335146 31493 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 9.
I20260805 22:25:08.296553 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:08.305011 3321 raft_consensus.cc:2726] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:25:08.332799 2956 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:08.328001 2842 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:08.346513 32594 raft_consensus.cc:1226] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 5. Current term is 6. Ops: []
W20260805 22:25:08.348414 31704 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56260 (request call id 2340) took 2348 ms (client timeout 1916 ms). Trace:
W20260805 22:25:08.348937 31704 rpcz_store.cc:269] 0805 22:25:05.999893 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.162618 (+162725us) service_pool.cc:227] Handling call
0805 22:25:08.347889 (+2185271us) spinlock_profiling.cc:243] Waited 744 ms on lock 0x7b54000ed154. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:08.348168 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1712053504}
I20260805 22:25:08.349255 3375 raft_consensus.cc:2726] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:08.355283 31492 raft_consensus.cc:2454] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
W20260805 22:25:08.363400 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 282632us
I20260805 22:25:08.366833 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:08.370172 31604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 2363) took 2387 ms. Trace:
I20260805 22:25:08.370653 31604 rpcz_store.cc:276] 0805 22:25:05.965326 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.420457 (+455131us) service_pool.cc:227] Handling call
0805 22:25:07.319153 (+898696us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:07.319712 (+ 559us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:08.349202 (+1029490us) spinlock_profiling.cc:243] Waited 724 ms on lock 0x7b540005d2d0. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731067a09 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:08.352426 (+ 3224us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1664092416}
I20260805 22:25:08.376858 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:08.377039 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:08.377204 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:08.378003 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:08.376897 3047 raft_consensus.cc:2726] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:08.382099 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:08.382486 3357 raft_consensus.cc:3032] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Stepping down as leader of term 1
I20260805 22:25:08.382925 3357 raft_consensus.cc:729] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:08.391703 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.392436 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.397603 3281 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:08.398319 3281 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:08.399165 31493 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:25:08.400501 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.392660 31490 raft_consensus.cc:1226] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260805 22:25:08.402827 3235 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:08.403296 3253 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:08.398073 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.404444 31486 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.LeaderStepDown from 127.0.0.1:54616 (request call id 20) took 2372 ms. Trace:
I20260805 22:25:08.404824 31486 rpcz_store.cc:276] 0805 22:25:06.032329 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:06.032552 (+ 223us) service_pool.cc:227] Handling call
0805 22:25:08.404117 (+2371565us) spinlock_profiling.cc:243] Waited 1.59 s on lock 0x7b54001fd954. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731062bc3 00007f2735fac81a 00007f272f69fd04 00007f272f69fc92 00007f272f69fbc1 00007f272f69fb3c 00007f272f69edf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:08.404239 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3662835968}
I20260805 22:25:08.405536 31704 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:25:08.408047 31708 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 2343) took 2407 ms. Trace:
I20260805 22:25:08.408283 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:08.397790 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.410118 31713 raft_consensus.cc:2354] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 8. Current term is 9.
W20260805 22:25:08.403221 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 379850us
W20260805 22:25:08.411780 2855 consensus_meta.cc:287] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.582s user 0.007s sys 0.011s
I20260805 22:25:08.412362 2855 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:08.412200 31708 rpcz_store.cc:276] 0805 22:25:06.000759 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:07.298844 (+1298085us) service_pool.cc:227] Handling call
0805 22:25:07.299199 (+ 355us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:07.789629 (+490430us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:08.407573 (+617944us) spinlock_profiling.cc:243] Waited 409 ms on lock 0x7b54001e2b54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:08.407856 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":940525952}
I20260805 22:25:08.414161 31604 raft_consensus.cc:1226] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 7. Current term is 8. Ops: []
I20260805 22:25:08.414336 3044 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:08.415014 31556 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:08.415138 3044 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:08.392590 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:08.426498 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:08.427062 31706 raft_consensus.cc:1226] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 4. Current term is 5. Ops: []
I20260805 22:25:08.423730 3235 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:08.429442 2855 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:08.429566 3044 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:08.427688 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.439939 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:08.440580 31493 raft_consensus.cc:1261] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 6 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260805 22:25:08.441905 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:08.442732 31501 raft_consensus.cc:3037] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:08.442929 31490 raft_consensus.cc:1261] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260805 22:25:08.444453 31708 raft_consensus.cc:1226] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:08.419169 3357 consensus_queue.cc:260] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:08.447149 3357 raft_consensus.cc:3037] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:25:08.456363 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:08.461004 31708 raft_consensus.cc:1226] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 7. Current term is 8. Ops: []
I20260805 22:25:08.462698 31501 raft_consensus.cc:2454] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:25:08.465646 31444 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:08.465708 3207 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:08.466955 3207 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:08.469269 3297 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260805 22:25:08.476923 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.525786 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.526260 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.527812 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.545629 31705 raft_consensus.cc:2379] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 7: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:08.546469 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:08.547598 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:08.548173 31705 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:08.549860 31713 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 10.
I20260805 22:25:08.528594 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.551090 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.551275 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.552107 31713 raft_consensus.cc:2396] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for term 5 because replica has last-logged OpId of term: 4 index: 3, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 2.
I20260805 22:25:08.552192 31704 raft_consensus.cc:2379] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 8: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:08.553051 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.456019 31556 leader_election.cc:398] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Vote denied by peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801) with higher term. Message: Invalid argument: T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5.
I20260805 22:25:08.530004 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:08.531419 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.557657 31714 raft_consensus.cc:2354] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:25:08.533478 31492 raft_consensus.cc:2396] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260805 22:25:08.537644 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:08.557678 2855 consensus_queue.cc:1046] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260805 22:25:08.539140 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:08.539392 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:08.559712 2855 raft_consensus.cc:3032] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:25:08.560103 2855 raft_consensus.cc:729] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:08.560341 31604 raft_consensus.cc:2354] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 8. Current term is 9.
I20260805 22:25:08.539901 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:08.542011 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:08.542299 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:08.543215 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.482370 3431 raft_consensus.cc:2726] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:25:08.544268 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:08.562944 32591 raft_consensus.cc:2379] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 6: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:08.528277 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.556406 31715 raft_consensus.cc:2379] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 6: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:08.566048 2855 consensus_queue.cc:260] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:08.567102 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.572845 31715 raft_consensus.cc:1226] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 6. Current term is 7. Ops: []
I20260805 22:25:08.573050 2855 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:08.571903 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:08.574761 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:08.576318 31708 raft_consensus.cc:2379] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 9: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:08.571102 31603 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:25:08.575870 3235 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:08.538318 3207 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:08.578131 31556 leader_election.cc:401] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Cancelling election due to peer responding with higher term
I20260805 22:25:08.577530 3197 consensus_queue.cc:1046] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 7.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260805 22:25:08.581337 3235 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:08.581837 3197 raft_consensus.cc:3032] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Stepping down as leader of term 7
I20260805 22:25:08.568375 31705 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 9.
I20260805 22:25:08.581987 3207 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:08.582325 3197 raft_consensus.cc:729] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
W20260805 22:25:08.593413 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 137884us
I20260805 22:25:08.596347 31597 raft_consensus.cc:1226] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 7. Current term is 8. Ops: [7.4-7.4]
I20260805 22:25:08.596113 3203 consensus_queue.cc:1046] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.006s
I20260805 22:25:08.597594 3431 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:08.601035 3197 consensus_queue.cc:260] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:08.602880 3197 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:08.607276 31556 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:08.608417 3442 consensus_queue.cc:1046] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:25:08.608845 31556 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:08.609606 3442 raft_consensus.cc:2726] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:08.610219 3442 raft_consensus.cc:3032] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:25:08.610692 3442 raft_consensus.cc:729] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:08.612202 3442 consensus_queue.cc:260] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:08.614223 3442 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:08.614846 3298 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:08.616564 3281 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:08.617133 3298 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:08.618562 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:08.619506 31711 raft_consensus.cc:1226] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 8. Current term is 9. Ops: []
I20260805 22:25:08.620304 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.621459 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:08.622259 31708 raft_consensus.cc:1261] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260805 22:25:08.622612 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:08.622730 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:08.623925 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.624017 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:08.624297 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:08.624678 31715 raft_consensus.cc:2379] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
W20260805 22:25:08.590786 31444 leader_election.cc:334] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Timed out: RequestConsensusVote RPC to 127.29.21.195:44831 timed out after 1.867s (SENT)
I20260805 22:25:08.623262 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:08.621008 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.621027 31709 raft_consensus.cc:2379] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 6: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:08.622259 31596 raft_consensus.cc:2379] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 11: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:08.623235 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:08.627388 31603 raft_consensus.cc:2379] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 9: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:08.629886 31713 raft_consensus.cc:2354] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 8. Current term is 9.
I20260805 22:25:08.631278 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:08.632110 31714 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:08.631346 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:08.634229 31486 raft_consensus.cc:2396] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 9 because replica has last-logged OpId of term: 8 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260805 22:25:08.597290 3203 raft_consensus.cc:3032] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:25:08.637614 31705 raft_consensus.cc:2454] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:25:08.638871 32581 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:60186
I20260805 22:25:08.639608 32581 raft_consensus.cc:595] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:08.639951 32581 raft_consensus.cc:603] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:25:08.641501 31444 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:08.637468 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:08.643020 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:08.636787 3203 raft_consensus.cc:729] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:08.654637 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:08.655488 31490 raft_consensus.cc:2379] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 4: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:08.610108 31359 catalog_manager.cc:5495] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:08.647712 32529 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:08.662535 31596 raft_consensus.cc:1261] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260805 22:25:08.654662 3357 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:25:08.666769 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.666169 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:08.675915 31705 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:25:08.678047 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:08.682099 31556 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:08.676461 31709 raft_consensus.cc:1226] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 9. Current term is 10. Ops: []
I20260805 22:25:08.685588 31708 raft_consensus.cc:2354] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5.
I20260805 22:25:08.680824 3288 consensus_queue.cc:1046] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.021s
I20260805 22:25:08.681802 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:08.687700 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:08.692535 31715 raft_consensus.cc:2354] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:25:08.694347 31492 raft_consensus.cc:1261] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260805 22:25:08.702087 31490 raft_consensus.cc:1261] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
I20260805 22:25:08.710469 31705 raft_consensus.cc:1261] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260805 22:25:08.710733 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:08.710947 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:08.704562 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.694531 3207 raft_consensus.cc:2781] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260805 22:25:08.715723 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 135376us
I20260805 22:25:08.719568 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.720280 31705 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:25:08.724354 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:08.725049 31501 raft_consensus.cc:3037] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:08.714308 3298 raft_consensus.cc:2726] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:25:08.727479 31709 raft_consensus.cc:3037] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
W20260805 22:25:08.732483 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 71545.9us
I20260805 22:25:08.734110 3203 consensus_queue.cc:260] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:08.719944 3281 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:08.718537 3207 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
W20260805 22:25:08.739437 31444 leader_election.cc:334] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Timed out: RequestConsensusVote RPC to 127.29.21.195:44831 timed out after 2.090s (SENT)
I20260805 22:25:08.740974 31501 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:25:08.743533 3207 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:08.735720 3357 raft_consensus.cc:3032] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:25:08.747248 3357 raft_consensus.cc:729] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:08.748291 3357 consensus_queue.cc:260] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:08.746264 31356 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34388 (request call id 280) took 11235 ms. Trace:
I20260805 22:25:08.752657 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.749866 31356 rpcz_store.cc:276] 0805 22:24:57.510419 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:24:57.510635 (+ 216us) service_pool.cc:227] Handling call
0805 22:24:57.717562 (+206927us) spinlock_profiling.cc:243] Waited 4.03 ms on lock 0x7b5400000910. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108d92 00007f27370597c9 00007f2736ff1c36 00007f2736fc2c28 00007f27370bcc3e 00007f27353cefb4 00007f27353cef42 00007f27353cee71 00007f27353cedec
0805 22:25:08.746040 (+11028478us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0805 22:25:04.321363 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0805 22:25:04.388359 (+ 66996us) write_op.cc:432] Acquiring schema lock in shared mode
0805 22:25:04.388400 (+ 41us) write_op.cc:435] Acquired schema lock
0805 22:25:04.388441 (+ 41us) tablet.cc:660] Decoding operations
0805 22:25:04.585450 (+197009us) write_op.cc:620] Acquiring the partition lock for write op
0805 22:25:04.591222 (+ 5772us) write_op.cc:641] Partition lock acquired for write op
0805 22:25:04.599082 (+ 7860us) tablet.cc:683] Acquiring locks for 9 operations
0805 22:25:05.213916 (+614834us) tablet.cc:699] Row locks acquired
0805 22:25:05.213952 (+ 36us) write_op.cc:260] PREPARE: finished
0805 22:25:05.316208 (+102256us) write_op.cc:270] Start()
0805 22:25:05.349740 (+ 33532us) write_op.cc:276] Timestamp: P: 1785968705306423 usec, L: 0
0805 22:25:05.365313 (+ 15573us) op_driver.cc:331] REPLICATION: starting
0805 22:25:05.459389 (+ 94076us) log.cc:842] Serialized 3632 byte log entry
0805 22:25:07.191415 (+1732026us) op_driver.cc:444] REPLICATION: finished
0805 22:25:07.671830 (+480415us) write_op.cc:301] APPLY: starting
0805 22:25:07.676754 (+ 4924us) tablet.cc:1365] starting BulkCheckPresence
0805 22:25:07.735076 (+ 58322us) tablet.cc:1368] finished BulkCheckPresence
0805 22:25:07.786270 (+ 51194us) tablet.cc:1370] starting ApplyRowOperation cycle
0805 22:25:07.787183 (+ 913us) tablet.cc:1381] finished ApplyRowOperation cycle
0805 22:25:07.787412 (+ 229us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0805 22:25:07.787452 (+ 40us) write_op.cc:312] APPLY: finished
0805 22:25:07.787738 (+ 286us) log.cc:842] Serialized 71 byte log entry
0805 22:25:07.787893 (+ 155us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:25:07.788021 (+ 128us) write_op.cc:454] Released schema lock
0805 22:25:07.788062 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0805 22:25:07.788337 (+ 275us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:25:07.788366 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":22931,"reported_tablets":33,"spinlock_wait_cycles":9265920,"child_traces":[["op",{"apply.queue_time_us":430586,"apply.run_cpu_time_us":3355,"apply.run_wall_time_us":130425,"num_ops":9,"prepare.queue_time_us":342055,"prepare.run_cpu_time_us":27770,"prepare.run_wall_time_us":1486984,"raft.queue_time_us":492757,"raft.run_cpu_time_us":15940,"raft.run_wall_time_us":412721,"replication_time_us":1810863,"thread_start_us":1175398,"threads_started":4,"wal-append.queue_time_us":493903}]]}
I20260805 22:25:08.744910 31556 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:08.759886 31493 raft_consensus.cc:2454] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 2.
I20260805 22:25:08.771512 31714 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 11.
I20260805 22:25:08.743742 3443 raft_consensus.cc:2781] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:25:08.759505 3357 raft_consensus.cc:3037] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:08.762296 3431 raft_consensus.cc:2726] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election lost for term 4. Reason: Vote denied by peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801) with higher term. Message: Invalid argument: T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5.
I20260805 22:25:08.754004 3203 raft_consensus.cc:3037] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:08.726969 31359 catalog_manager.cc:5495] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 6 to 8, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 8 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:25:08.745157 31444 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:08.798893 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:08.800449 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:08.807003 31556 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:08.808347 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:25:08.810072 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 84249.2us
I20260805 22:25:08.810262 31704 raft_consensus.cc:1261] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:25:08.810504 3201 raft_consensus.cc:2726] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260805 22:25:08.810761 31444 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.195:44831 timed out after 2.110s (SENT) [suppressed 37 similar messages]
I20260805 22:25:08.812009 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: true dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:08.812582 3446 raft_consensus.cc:2726] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:25:08.813165 31596 raft_consensus.cc:1226] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 7. Current term is 8. Ops: []
I20260805 22:25:08.813491 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.813194 31490 raft_consensus.cc:1226] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5. Ops: [4.6-4.6]
I20260805 22:25:08.815371 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.816845 31667 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:08.819118 31359 catalog_manager.cc:5495] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 3 to 4, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 4 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:08.816960 31708 raft_consensus.cc:1261] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260805 22:25:08.788034 3443 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:08.823648 3443 raft_consensus.cc:3037] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:08.825673 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:08.812601 31594 raft_consensus.cc:1226] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 1. Current term is 2. Ops: []
I20260805 22:25:08.823268 3442 raft_consensus.cc:2781] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:25:08.827360 3442 raft_consensus.cc:686] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:08.828495 3442 consensus_queue.cc:237] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:08.831707 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:08.831938 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.820796 31359 catalog_manager.cc:5495] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 3 to 4, leader changed from 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 4 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20260805 22:25:08.812026 31444 leader_election.cc:334] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831): Timed out: RequestConsensusVote RPC to 127.29.21.195:44831 timed out after 2.213s (SENT)
I20260805 22:25:08.844851 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:08.844285 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:08.851248 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:08.852128 31492 raft_consensus.cc:2379] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 8: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:08.845551 31706 raft_consensus.cc:2362] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 11. Re-sending same reply.
I20260805 22:25:08.859138 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:08.856264 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:08.868760 31594 raft_consensus.cc:2379] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 8: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:08.872404 31602 raft_consensus.cc:1261] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260805 22:25:08.873445 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:08.873754 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 230632us
W20260805 22:25:08.876900 32529 leader_election.cc:334] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Timed out: RequestConsensusVote RPC to 127.29.21.194:41897 timed out after 1.507s (SENT)
I20260805 22:25:08.877740 3144 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260805 22:25:08.877285 32529 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:08.894325 31490 raft_consensus.cc:1226] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 7. Current term is 8. Ops: []
I20260805 22:25:08.894644 3025 raft_consensus.cc:2726] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:25:08.899075 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:08.900063 31704 raft_consensus.cc:1261] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 4 index: 7. Preceding OpId from leader: term: 5 index: 8. (index mismatch)
W20260805 22:25:08.905699 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 93714.3us
I20260805 22:25:08.906764 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:08.914376 3298 raft_consensus.cc:2726] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:08.908624 3050 consensus_queue.cc:1046] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260805 22:25:08.917989 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:08.919147 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:08.920490 31604 raft_consensus.cc:2454] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:25:08.861876 3207 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:08.923691 31597 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:25:08.929800 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:08.932782 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:08.941042 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 124456us
I20260805 22:25:08.941716 3231 raft_consensus.cc:3032] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Stepping down as leader of term 7
I20260805 22:25:08.942238 3231 raft_consensus.cc:729] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:08.943207 3231 consensus_queue.cc:260] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:08.944649 3231 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:08.942070 31667 leader_election.cc:398] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 8. Current term is 9.
I20260805 22:25:08.950516 31667 leader_election.cc:401] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Cancelling election due to peer responding with higher term
I20260805 22:25:08.958715 31359 catalog_manager.cc:5495] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 2 to 7, leader changed from 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 7 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:25:08.960577 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:08.965706 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:08.973984 3443 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:08.955953 3411 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:08.977135 3411 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:08.979310 3411 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:08.984378 3281 consensus_queue.cc:1035] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:25:08.985329 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:08.986270 3460 log.cc:924] Time spent T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.057s user 0.000s sys 0.000s
I20260805 22:25:08.988917 3207 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:08.969470 3297 consensus_queue.cc:1046] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.022s
I20260805 22:25:08.993161 3297 raft_consensus.cc:3032] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Stepping down as leader of term 7
I20260805 22:25:09.010633 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:09.008522 31709 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:25:09.013936 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 9 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:09.017628 31667 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:09.017778 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
W20260805 22:25:09.030301 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 153433us
I20260805 22:25:09.030969 31602 raft_consensus.cc:1261] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260805 22:25:09.037930 31603 raft_consensus.cc:1226] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 8. Current term is 9. Ops: []
I20260805 22:25:09.038763 31597 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:09.008314 3443 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:09.011943 3297 raft_consensus.cc:729] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:09.043139 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:09.040318 31359 catalog_manager.cc:5495] T dcea14dac2934d139e2c5ab6d0b8648d P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 2 to 5, leader changed from 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 5 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:25:09.052954 2762 consensus_queue.cc:1046] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:25:09.053306 31667 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:09.054113 2762 raft_consensus.cc:3032] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 LEADER]: Stepping down as leader of term 7
I20260805 22:25:09.054124 3025 consensus_queue.cc:1035] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:25:09.055250 2762 raft_consensus.cc:729] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:09.056380 2762 consensus_queue.cc:260] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:09.057397 3351 raft_consensus.cc:2781] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:25:09.057499 3345 raft_consensus.cc:2726] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election lost for term 8. Reason: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 8. Current term is 9.
I20260805 22:25:09.057992 3351 raft_consensus.cc:686] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:09.058714 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:09.058977 3351 consensus_queue.cc:237] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:09.061484 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:09.062315 31492 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:09.063488 3201 consensus_queue.cc:1035] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:25:09.057976 2762 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:09.070010 32589 raft_consensus.cc:1226] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 6. Current term is 7. Ops: [6.7-6.7]
I20260805 22:25:09.066491 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:09.071370 31594 raft_consensus.cc:2379] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 4: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:09.076231 2956 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election won for term 8
I20260805 22:25:09.076009 3281 consensus_queue.cc:1046] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:25:09.090632 2956 raft_consensus.cc:686] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [term 8 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:09.100100 31604 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
I20260805 22:25:09.105218 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:09.146275 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:09.143649 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:09.142457 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:09.149457 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:09.142033 32589 raft_consensus.cc:1261] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:25:09.141706 31359 catalog_manager.cc:5495] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 3 to 4, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 4 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:09.112325 31492 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:25:09.177459 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:09.145658 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:09.179410 31359 catalog_manager.cc:5495] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 4 to 6. New cstate: current_term: 6 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:09.161475 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:09.123303 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:09.181142 31604 raft_consensus.cc:2379] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 6: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:09.178030 31490 raft_consensus.cc:2379] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 7: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:09.145239 32529 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.184330 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:09.185534 31602 raft_consensus.cc:2354] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 7. Current term is 8.
W20260805 22:25:09.192111 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 46960.3us
I20260805 22:25:09.194490 31359 catalog_manager.cc:5495] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:09.142845 2956 consensus_queue.cc:237] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 7.6, Last appended by leader: 6, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:09.203519 32529 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.195451 31359 catalog_manager.cc:5495] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 reported cstate change: term changed from 3 to 5, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 5 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:09.144795 3203 consensus_queue.cc:1035] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.057s
I20260805 22:25:09.212069 31495 raft_consensus.cc:2379] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 6: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:09.146291 3298 consensus_queue.cc:1035] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.049s
I20260805 22:25:09.221765 3144 raft_consensus.cc:2726] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:25:09.224236 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:09.228752 31444 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.229648 3253 raft_consensus.cc:2726] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260805 22:25:09.230257 31444 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.231801 31444 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.232728 3253 raft_consensus.cc:2726] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:09.235987 31444 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.236972 3253 raft_consensus.cc:2726] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:09.237238 31444 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.243798 31597 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
I20260805 22:25:09.232842 3357 raft_consensus.cc:2726] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:09.240283 31501 raft_consensus.cc:2454] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
W20260805 22:25:09.246083 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 50238.8us
I20260805 22:25:09.252233 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:09.253499 31444 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.145714 3297 consensus_queue.cc:260] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:09.238008 3303 consensus_queue.cc:1035] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.014s
I20260805 22:25:09.253371 3453 raft_consensus.cc:2726] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:25:09.275234 31667 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:09.249270 3253 raft_consensus.cc:2726] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:25:09.279112 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:09.305368 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 57082.8us
I20260805 22:25:09.258034 3357 raft_consensus.cc:2726] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:25:09.248333 3144 consensus_queue.cc:1046] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 7.9, Next index: 10, Last known committed idx: 8, Time since last communication: 0.026s
I20260805 22:25:09.251392 2842 consensus_queue.cc:1046] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.028s
W20260805 22:25:09.311154 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 45645us
I20260805 22:25:09.314119 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:09.314915 31667 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:09.314942 3328 raft_consensus.cc:3032] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Stepping down as leader of term 7
I20260805 22:25:09.315543 3328 raft_consensus.cc:729] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:09.316562 3328 consensus_queue.cc:260] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:09.317050 31667 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:09.312125 2855 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:09.312669 31444 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.318069 2855 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:09.318559 3321 raft_consensus.cc:3037] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:09.319484 31444 leader_election.cc:398] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 8. Current term is 9.
I20260805 22:25:09.320495 2855 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:09.320293 3197 consensus_queue.cc:1035] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260805 22:25:09.312425 3201 consensus_queue.cc:1046] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:25:09.321621 31444 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.322381 3328 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:09.322906 3345 raft_consensus.cc:2726] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:09.324658 31667 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:09.325745 3025 raft_consensus.cc:2726] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260805 22:25:09.325970 31667 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:09.328784 3281 raft_consensus.cc:2726] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:09.328369 31444 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:09.329742 2855 raft_consensus.cc:2726] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260805 22:25:09.317708 3272 raft_consensus.cc:2741] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election decision vote started in defunct term 4: won
I20260805 22:25:09.328888 31667 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:09.331102 31444 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:09.332010 3281 raft_consensus.cc:2741] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election decision vote started in defunct term 5: won
I20260805 22:25:09.332100 31667 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:09.331508 3272 raft_consensus.cc:2726] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260805 22:25:09.333824 31444 leader_election.cc:398] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:25:09.334194 31444 leader_election.cc:401] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 election: Cancelling election due to peer responding with higher term
I20260805 22:25:09.334822 2855 raft_consensus.cc:2781] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:25:09.335389 2855 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:09.335827 2855 raft_consensus.cc:3037] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:09.339277 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:09.322010 2452 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260805 22:25:09.349509 3232 raft_consensus.cc:2726] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader election lost for term 3. Reason: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4.
I20260805 22:25:09.350363 3361 consensus_queue.cc:1035] T 94d0d26905694943b7135a9b805cf106 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.008s
I20260805 22:25:09.352097 31444 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:09.354293 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:09.330523 2842 raft_consensus.cc:2781] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:25:09.361852 31709 raft_consensus.cc:2379] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 9: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:09.321681 3142 consensus_queue.cc:1035] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.047s
I20260805 22:25:09.382306 31492 raft_consensus.cc:1226] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:09.385319 31444 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.382803 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:09.386554 31444 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.343415 2452 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
W20260805 22:25:09.389180 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 46644.1us
I20260805 22:25:09.390758 31486 raft_consensus.cc:1226] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:09.393795 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:09.397912 31667 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:09.362839 2842 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:09.399073 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:09.401126 31704 raft_consensus.cc:1226] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:09.399672 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:09.403924 31444 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.385478 3253 raft_consensus.cc:2781] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260805 22:25:09.402355 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 178041us
I20260805 22:25:09.411406 31554 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.393167 3281 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:09.419523 31486 raft_consensus.cc:1261] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260805 22:25:09.393389 2452 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:09.422832 31704 raft_consensus.cc:1261] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260805 22:25:09.404029 2842 raft_consensus.cc:3037] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:09.418026 3201 consensus_queue.cc:1035] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20260805 22:25:09.414042 3050 raft_consensus.cc:2726] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:09.411511 3361 consensus_queue.cc:1035] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.023s
I20260805 22:25:09.412418 3253 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:09.415292 3197 raft_consensus.cc:2726] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260805 22:25:09.438508 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 48233.4us
I20260805 22:25:09.440207 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:09.447055 31486 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 10.
I20260805 22:25:09.425354 3442 raft_consensus.cc:2726] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260805 22:25:09.452517 31667 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:09.455888 31495 raft_consensus.cc:1226] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 6. Current term is 7. Ops: []
I20260805 22:25:09.465417 2855 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:09.430495 2956 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260805 22:25:09.437826 3429 consensus_queue.cc:1046] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 5, Time since last communication: 0.034s
I20260805 22:25:09.467726 3321 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:25:09.468101 3288 raft_consensus.cc:3032] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 8 LEADER]: Stepping down as leader of term 8
I20260805 22:25:09.472069 3429 raft_consensus.cc:3032] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:25:09.472301 3288 raft_consensus.cc:729] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:09.472514 3429 raft_consensus.cc:729] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:09.473285 3288 consensus_queue.cc:260] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:09.474136 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 44604.7us
I20260805 22:25:09.474647 3288 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:09.474967 31444 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.475939 3321 raft_consensus.cc:2726] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:09.476315 31444 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.477202 3321 raft_consensus.cc:2726] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260805 22:25:09.428097 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:25:09.468360 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 62535.3us
I20260805 22:25:09.469617 3303 raft_consensus.cc:3032] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 7 LEADER]: Stepping down as leader of term 7
I20260805 22:25:09.479686 3303 raft_consensus.cc:729] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 7 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:09.480643 3303 consensus_queue.cc:260] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 7.9, Last appended by leader: 9, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:09.482072 3303 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:09.484031 3297 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:09.488730 2855 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:09.489837 31601 raft_consensus.cc:1226] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 8. Current term is 9. Ops: []
I20260805 22:25:09.454608 3446 consensus_queue.cc:1035] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.040s
I20260805 22:25:09.439502 3253 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:09.502353 31554 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:09.504932 3361 raft_consensus.cc:2781] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:25:09.505575 3361 raft_consensus.cc:686] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:09.507015 3361 consensus_queue.cc:237] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:09.513088 31495 raft_consensus.cc:1261] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260805 22:25:09.519604 3446 raft_consensus.cc:2726] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:09.526593 31554 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:09.527921 31554 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:09.528694 3443 raft_consensus.cc:2726] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:09.528805 31554 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
W20260805 22:25:09.520133 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 67714.3us
I20260805 22:25:09.529968 31667 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:09.530905 31667 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:09.531553 2762 raft_consensus.cc:2726] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260805 22:25:09.532213 2762 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:25:09.533209 3443 raft_consensus.cc:2781] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:25:09.535161 31667 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:09.520860 3429 consensus_queue.cc:260] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:09.537170 3429 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:09.547257 31554 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:09.587388 31444 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf
W20260805 22:25:09.595938 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 91358.9us
I20260805 22:25:09.599833 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:09.580489 3442 raft_consensus.cc:2781] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 2 FOLLOWER]: Leader election won for term 2
I20260805 22:25:09.584591 3025 raft_consensus.cc:2781] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:25:09.604719 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:09.606098 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:09.603657 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:09.615253 31708 raft_consensus.cc:1261] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
I20260805 22:25:09.618130 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:09.619019 31704 raft_consensus.cc:2379] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 7: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:09.605618 31706 raft_consensus.cc:2362] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 9. Re-sending same reply.
I20260805 22:25:09.600795 3231 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election won for term 8
I20260805 22:25:09.590060 3446 consensus_queue.cc:1046] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 2.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.041s
I20260805 22:25:09.638675 2452 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:09.639411 31501 raft_consensus.cc:1261] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 7 index: 4. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260805 22:25:09.640079 3253 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:09.640478 3345 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:09.641433 31667 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:09.653977 31667 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:09.632256 3025 raft_consensus.cc:686] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:09.656965 3231 raft_consensus.cc:686] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 8 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
W20260805 22:25:09.682973 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 216630us
I20260805 22:25:09.692633 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:09.694363 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:09.695509 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:09.695842 32589 raft_consensus.cc:1261] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260805 22:25:09.696933 31601 raft_consensus.cc:1226] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:09.697732 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:09.699225 31603 raft_consensus.cc:1261] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260805 22:25:09.700397 31596 raft_consensus.cc:1226] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 3. Current term is 4. Ops: []
I20260805 22:25:09.694141 3491 raft_consensus.cc:2726] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260805 22:25:09.706333 3025 consensus_queue.cc:237] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:09.657239 3345 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:09.718174 3493 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260805 22:25:09.719228 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 280186us
I20260805 22:25:09.711925 2956 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
W20260805 22:25:09.726796 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 179623us
I20260805 22:25:09.732266 31554 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:09.748605 2452 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:09.771167 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 139542us
I20260805 22:25:09.801781 32529 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:09.802992 3142 raft_consensus.cc:2726] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260805 22:25:09.803921 3484 log.cc:924] Time spent T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.114s user 0.000s sys 0.000s
I20260805 22:25:09.804471 32529 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.806288 3142 raft_consensus.cc:2726] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:09.807044 32529 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:09.813536 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 81367.5us
I20260805 22:25:09.715673 2842 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:09.790940 3443 raft_consensus.cc:686] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [term 6 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
W20260805 22:25:09.844901 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 55915.9us
W20260805 22:25:09.848086 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 194175us
I20260805 22:25:09.848822 31667 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:09.812328 3442 raft_consensus.cc:686] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [term 2 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:09.854357 31495 raft_consensus.cc:1226] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5. Ops: []
I20260805 22:25:09.839793 3050 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:09.847335 3253 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:09.846350 3298 raft_consensus.cc:2726] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
W20260805 22:25:09.905742 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 56170.7us
I20260805 22:25:09.809685 3231 consensus_queue.cc:237] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.9, Last appended by leader: 9, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:09.908931 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 101951us
I20260805 22:25:09.879031 3505 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:25:09.918517 3525 log.cc:924] Time spent T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.068s user 0.001s sys 0.000s
I20260805 22:25:09.918938 3497 raft_consensus.cc:2726] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260805 22:25:09.920713 31423 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica feb5e7ab4b1541beaf2674c3c5500385 is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:09.921164 3442 consensus_queue.cc:237] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:09.922880 3487 log.cc:924] Time spent T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.263s user 0.003s sys 0.000s
W20260805 22:25:09.924451 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 70941.4us
I20260805 22:25:09.925997 3505 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:09.864763 3025 consensus_queue.cc:1035] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.046s
I20260805 22:25:09.928025 3505 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:09.910229 3050 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:09.929126 3493 raft_consensus.cc:3032] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:25:09.929541 3493 raft_consensus.cc:729] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
W20260805 22:25:09.931690 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 85962.6us
W20260805 22:25:09.935181 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 86437us
I20260805 22:25:09.936774 32529 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.936883 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:09.930388 3493 consensus_queue.cc:260] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:09.940788 31667 leader_election.cc:398] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 7. Current term is 8.
I20260805 22:25:09.941159 31667 leader_election.cc:401] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Cancelling election due to peer responding with higher term
I20260805 22:25:09.941579 31444 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:09.944468 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:09.945106 32589 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:25:09.930464 3050 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:09.938138 3321 consensus_queue.cc:1035] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.008s
I20260805 22:25:09.931191 3272 raft_consensus.cc:2726] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:25:09.951894 2956 raft_consensus.cc:2726] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:09.953882 2842 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:09.955917 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 527d60ab6db74595bed7e87ca2ecc6e8, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:09.956314 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 527d60ab6db74595bed7e87ca2ecc6e8, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:10.008222 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 96685.6us
I20260805 22:25:10.022575 3345 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:10.022970 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 87048.2us
I20260805 22:25:10.024346 3429 consensus_queue.cc:1046] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 5.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:25:09.969693 2956 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:09.973780 3272 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:10.035815 3345 consensus_queue.cc:1046] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 5.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260805 22:25:10.037052 3345 raft_consensus.cc:3032] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Stepping down as leader of term 5
W20260805 22:25:10.037369 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 66851.2us
I20260805 22:25:10.038076 31668 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:10.037425 3345 raft_consensus.cc:729] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:09.943625 3443 consensus_queue.cc:237] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.025346 3499 consensus_queue.cc:1035] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:25:10.040380 2762 consensus_queue.cc:1035] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:25:10.041589 3493 raft_consensus.cc:3037] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:10.039400 3345 consensus_queue.cc:260] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:10.043917 3345 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:10.028407 3503 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
W20260805 22:25:10.029033 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 5e4ad7072aab4331878e988a26a6d887, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:10.059278 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 5e4ad7072aab4331878e988a26a6d887, TS feb5e7ab4b1541beaf2674c3c5500385 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:09.973644 3297 raft_consensus.cc:2726] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260805 22:25:10.025074 31554 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:10.017670 3477 raft_consensus.cc:3037] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:10.002609 3446 consensus_queue.cc:1046] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 5.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.029s
I20260805 22:25:10.035630 3500 raft_consensus.cc:3032] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:25:10.044411 3505 consensus_queue.cc:1046] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 3.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260805 22:25:10.046530 2956 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.034168 32529 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:10.026183 31444 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:10.064481 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 97579us
I20260805 22:25:10.082388 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:10.084036 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:10.079068 31667 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:10.089076 31596 raft_consensus.cc:1261] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 5 index: 6. (index mismatch)
I20260805 22:25:10.090428 31709 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:10.091812 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:10.078693 3506 raft_consensus.cc:2781] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:10.096127 31667 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:10.061992 3351 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:10.087358 3328 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:10.079248 3499 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260805 22:25:10.085112 3500 raft_consensus.cc:729] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:10.101171 31554 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:10.108286 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:10.111563 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:10.112257 31495 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:25:10.090262 3361 consensus_queue.cc:1035] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20260805 22:25:10.098372 3506 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:10.114657 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:10.101976 3499 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:10.120762 31495 raft_consensus.cc:2379] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 2: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:10.094274 3429 consensus_queue.cc:1035] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.021s
I20260805 22:25:10.105909 2842 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:10.105060 3298 raft_consensus.cc:2781] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:25:10.077930 3503 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.115911 3505 raft_consensus.cc:2726] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:25:10.114914 2855 raft_consensus.cc:2726] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:10.092170 3496 consensus_queue.cc:1046] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.031s
W20260805 22:25:10.124653 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 90649.2us
W20260805 22:25:10.133657 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 107548us
I20260805 22:25:10.128475 31554 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:10.119114 3506 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:10.101644 3351 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:10.139834 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 51436.6us
I20260805 22:25:10.149238 3351 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:10.145018 3496 raft_consensus.cc:3032] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:25:10.150038 3496 raft_consensus.cc:729] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:10.150475 3503 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:10.151476 3519 log.cc:924] Time spent T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260805 22:25:10.151810 3272 raft_consensus.cc:2726] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election lost for term 7. Reason: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 7. Current term is 8.
I20260805 22:25:10.122834 3499 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:10.105448 3328 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.143199 31609 raft_consensus.cc:1261] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
W20260805 22:25:10.153107 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 57067.5us
I20260805 22:25:10.143901 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:10.144577 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:10.144917 2956 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:10.145236 32529 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:10.150904 3496 consensus_queue.cc:260] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.154240 31596 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 9.
I20260805 22:25:10.123247 3527 raft_consensus.cc:2726] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:10.131628 3298 raft_consensus.cc:686] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:10.159963 3303 raft_consensus.cc:2726] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:10.166072 3298 consensus_queue.cc:237] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:10.169416 3328 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:10.170297 31486 raft_consensus.cc:1261] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 7 index: 6. Preceding OpId from leader: term: 8 index: 7. (index mismatch)
I20260805 22:25:10.160616 32529 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:10.174206 3303 raft_consensus.cc:2726] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:25:10.161365 3496 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:10.156960 3446 raft_consensus.cc:2726] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260805 22:25:10.158668 3505 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:10.163769 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:10.137185 3500 consensus_queue.cc:260] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 5.4, Last appended by leader: 4, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.178068 3505 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.179788 3500 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:10.180120 3505 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:10.185571 3491 raft_consensus.cc:3032] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Stepping down as leader of term 3
I20260805 22:25:10.185985 3491 raft_consensus.cc:729] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:10.186813 3491 consensus_queue.cc:260] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:10.187467 3509 log.cc:924] Time spent T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.089s user 0.000s sys 0.001s
I20260805 22:25:10.196766 32589 raft_consensus.cc:1261] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260805 22:25:10.204177 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 44461.7us
I20260805 22:25:10.205328 31444 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:10.207031 31709 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
W20260805 22:25:10.208334 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 79958.2us
I20260805 22:25:10.209990 32529 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:10.213644 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:10.211830 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:10.211841 31667 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:10.225579 31486 raft_consensus.cc:1261] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:25:10.225958 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:10.227120 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:10.227995 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:10.237787 31667 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:10.226509 3272 consensus_queue.cc:1035] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.015s
I20260805 22:25:10.242717 3499 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.244832 3499 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 11 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:10.215065 3297 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260805 22:25:10.245978 3297 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:10.246363 3297 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:10.213613 3491 raft_consensus.cc:3037] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:25:10.252609 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 4deaa72a885a43bfbc6171a7ed13ad81, TS 516cb07ce3064ccbbca35e912d62816c -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:10.252964 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 4deaa72a885a43bfbc6171a7ed13ad81, TS 516cb07ce3064ccbbca35e912d62816c -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.245726 31601 raft_consensus.cc:1226] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 10. Current term is 11. Ops: [10.7-10.7]
I20260805 22:25:10.256071 3361 raft_consensus.cc:3037] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 1 FOLLOWER]: Advancing to term 2
I20260805 22:25:10.259855 3321 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:10.260460 3321 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:10.255245 31554 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:10.265247 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 54133.5us
I20260805 22:25:10.265973 31444 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:10.241700 3529 log.cc:924] Time spent T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.088s user 0.000s sys 0.000s
I20260805 22:25:10.239147 3303 raft_consensus.cc:2726] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260805 22:25:10.273265 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 17b3240dd1b947178dafdc9cac7f1e5e, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:10.273558 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 73382.8us
I20260805 22:25:10.256001 3477 raft_consensus.cc:2726] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:25:10.274453 32529 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:10.275719 32529 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:10.277297 32594 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
mode: GRACEFUL
new_leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
from {username='slave'} at 127.0.0.1:60186
I20260805 22:25:10.273581 3503 raft_consensus.cc:2726] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260805 22:25:10.273729 3357 raft_consensus.cc:2726] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260805 22:25:10.278000 32594 raft_consensus.cc:595] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Received request to transfer leadership to TS feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:10.278133 3321 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:10.278311 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:10.278373 32594 raft_consensus.cc:603] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:25:10.282677 3442 raft_consensus.cc:3032] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:25:10.283110 3442 raft_consensus.cc:729] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:10.284178 3303 raft_consensus.cc:2726] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:10.283805 3297 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.284844 3303 raft_consensus.cc:2726] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260805 22:25:10.284614 3506 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:10.286854 3297 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:10.287182 3321 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:25:10.273676 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 17b3240dd1b947178dafdc9cac7f1e5e, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:10.289680 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 54101.7us
I20260805 22:25:10.291623 31554 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:10.294770 31554 leader_election.cc:398] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5.
I20260805 22:25:10.294999 3442 consensus_queue.cc:260] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.295569 31707 raft_consensus.cc:1226] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260805 22:25:10.296293 3442 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:10.296633 3361 raft_consensus.cc:2726] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260805 22:25:10.296931 3429 raft_consensus.cc:2726] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260805 22:25:10.297686 3429 consensus_queue.cc:1046] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 9.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:25:10.299095 3492 consensus_queue.cc:1035] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260805 22:25:10.300031 3443 raft_consensus.cc:3032] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 9 LEADER]: Stepping down as leader of term 9
I20260805 22:25:10.300402 3443 raft_consensus.cc:729] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:10.300800 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:10.301435 31714 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 9.
I20260805 22:25:10.302868 31554 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:10.287691 3321 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.308234 3544 consensus_queue.cc:1035] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.076s
I20260805 22:25:10.308478 31444 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:10.286932 3506 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:10.320441 3351 consensus_queue.cc:1046] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 8.6, Next index: 7, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:25:10.320809 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:10.315778 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:10.315374 31486 raft_consensus.cc:1261] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260805 22:25:10.320317 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:10.322067 31494 raft_consensus.cc:1226] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 5. Current term is 6. Ops: [5.6-5.6]
I20260805 22:25:10.330490 31499 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:25:10.322223 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:10.322768 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:10.331357 31490 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 7
I20260805 22:25:10.331447 32589 raft_consensus.cc:3037] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:10.333529 31667 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:10.336323 31603 raft_consensus.cc:1261] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 5 index: 4. Preceding OpId from leader: term: 6 index: 5. (index mismatch)
I20260805 22:25:10.315424 3549 raft_consensus.cc:2726] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:10.344949 31667 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:10.346632 31667 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:10.322741 3493 raft_consensus.cc:2726] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260805 22:25:10.321828 3357 raft_consensus.cc:2726] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:10.327495 3547 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:10.325927 2842 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:10.326002 3298 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:10.349079 31667 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:10.352052 31554 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:10.342584 2956 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:10.371292 3560 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:10.371912 3547 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:10.372793 3490 consensus_queue.cc:1035] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.008s
I20260805 22:25:10.373273 32579 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 7 index: 9. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260805 22:25:10.360041 3361 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:10.360355 3272 consensus_queue.cc:1035] T dc38ff59fd654f6689e3a88fdbb765cf P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.014s
I20260805 22:25:10.345503 3492 consensus_queue.cc:1035] T fb8dc633587f48d1bba2a54a1619bc3e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.029s
I20260805 22:25:10.342839 3446 consensus_queue.cc:1035] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.031s
I20260805 22:25:10.366727 2842 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.301218 3443 consensus_queue.cc:260] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 9.4, Last appended by leader: 4, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.375432 3025 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:10.375640 3298 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:10.376571 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 68183us
I20260805 22:25:10.346325 2452 raft_consensus.cc:2726] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:10.362008 3491 raft_consensus.cc:2726] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:10.379410 31554 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:10.390225 31490 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:10.387866 3560 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:10.387658 3496 raft_consensus.cc:2726] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260805 22:25:10.422299 31667 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:10.425513 31444 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:10.409106 3231 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.018s
I20260805 22:25:10.403156 3506 consensus_queue.cc:1035] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
W20260805 22:25:10.429214 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 49881.5us
I20260805 22:25:10.429486 31444 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:10.430516 31554 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:10.398526 2956 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:10.432742 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:10.434126 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:10.435753 31594 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Advancing to term 10
W20260805 22:25:10.429160 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 100186us
I20260805 22:25:10.435846 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:10.436528 31604 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
I20260805 22:25:10.440845 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:10.441594 31600 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
I20260805 22:25:10.447311 31444 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:10.428493 3328 raft_consensus.cc:2781] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:25:10.456909 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:10.424940 3560 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:10.433180 2452 raft_consensus.cc:2781] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:25:10.433367 3571 raft_consensus.cc:2726] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:25:10.434566 3513 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:10.473999 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:10.464720 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:10.466650 31709 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:10.438277 3496 raft_consensus.cc:2726] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election lost for term 4. Reason: could not achieve majority
I20260805 22:25:10.480608 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:10.437319 3050 raft_consensus.cc:2726] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:25:10.441598 3443 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:10.480050 31711 raft_consensus.cc:1261] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260805 22:25:10.481191 31597 raft_consensus.cc:1226] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 7. Current term is 8. Ops: [7.9-7.9]
I20260805 22:25:10.422960 3025 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.484906 31594 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 10.
I20260805 22:25:10.485929 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:10.488639 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 51904us
I20260805 22:25:10.493700 31713 raft_consensus.cc:1261] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20260805 22:25:10.468639 3272 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:10.437582 3429 consensus_queue.cc:1035] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.049s
I20260805 22:25:10.504061 32594 raft_consensus.cc:1261] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
W20260805 22:25:10.503762 32529 leader_election.cc:334] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897): Timed out: RequestConsensusVote RPC to 127.29.21.194:41897 timed out after 2.105s (SENT)
I20260805 22:25:10.497174 3571 consensus_queue.cc:1046] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 4.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.009s
I20260805 22:25:10.483641 3321 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:10.491941 3492 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:10.483210 3328 raft_consensus.cc:686] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 6 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:10.483305 3493 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:10.510107 31709 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 10.
I20260805 22:25:10.520149 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:10.521937 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:10.507472 3571 raft_consensus.cc:3032] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:25:10.524793 31600 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:10.525019 3571 raft_consensus.cc:729] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:10.525972 3571 consensus_queue.cc:260] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.527392 3571 raft_consensus.cc:3037] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:10.528692 32589 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:10.488124 3574 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:10.529439 3574 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:10.531615 3574 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:10.488373 2452 raft_consensus.cc:686] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:10.533562 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 4
I20260805 22:25:10.522476 3577 consensus_queue.cc:1046] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.011s
I20260805 22:25:10.534634 3577 raft_consensus.cc:3032] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:25:10.535009 3577 raft_consensus.cc:729] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:10.522997 32529 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:10.507632 3231 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:10.533130 2452 consensus_queue.cc:237] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:10.536851 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:10.537161 31667 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:10.499130 3050 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:10.505053 3232 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:10.571660 32594 raft_consensus.cc:2379] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 7: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
W20260805 22:25:10.576857 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 73119.4us
I20260805 22:25:10.525336 3547 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:10.488445 3513 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:10.575068 3232 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:10.505360 3272 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.570933 3050 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.571513 3025 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:10.591759 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 144521us
I20260805 22:25:10.600222 31444 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:10.604538 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:10.605741 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:10.606441 31493 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:25:10.607980 32529 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:10.539356 3493 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.596511 2842 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:10.617792 3605 raft_consensus.cc:2781] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:10.567108 3231 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.618173 3446 consensus_queue.cc:1035] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:25:10.620117 3513 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:10.621747 3272 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:10.622799 3579 log.cc:924] Time spent T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.087s user 0.001s sys 0.000s
I20260805 22:25:10.623250 3298 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:10.618027 3582 raft_consensus.cc:2726] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:25:10.623297 3050 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:10.628057 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:10.647178 32594 raft_consensus.cc:2396] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260805 22:25:10.588479 3429 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:10.629559 3605 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:10.599308 3232 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:10.576375 3492 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.621747 3550 raft_consensus.cc:2781] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:10.637948 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:10.701223 3582 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:10.701957 3527 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
W20260805 22:25:10.699256 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 217527us
W20260805 22:25:10.703832 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 58024.5us
I20260805 22:25:10.713938 31444 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:10.703583 31709 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:25:10.716806 31554 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:10.583987 3577 consensus_queue.cc:260] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:10.696870 3605 raft_consensus.cc:3037] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:10.725090 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:10.730926 31601 raft_consensus.cc:2379] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 8: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:10.726164 31493 raft_consensus.cc:1261] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20260805 22:25:10.697856 2956 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:10.609346 3328 consensus_queue.cc:237] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 5, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:10.707800 3550 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:10.774780 3550 raft_consensus.cc:3037] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:10.716609 3582 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:10.778000 3580 consensus_queue.cc:1046] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.017s
I20260805 22:25:10.781615 3591 consensus_queue.cc:1035] T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.050s
W20260805 22:25:10.782167 3600 log.cc:924] Time spent T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.062s user 0.000s sys 0.000s
I20260805 22:25:10.775130 3598 raft_consensus.cc:2726] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Leader election lost for term 11. Reason: could not achieve majority
I20260805 22:25:10.705209 3429 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.793722 3587 raft_consensus.cc:2781] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:10.799716 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:25:10.801486 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 66908.8us
I20260805 22:25:10.720315 3527 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:10.783696 3446 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:25:10.860333 3597 log.cc:924] Time spent T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.065s user 0.000s sys 0.001s
I20260805 22:25:10.884094 3577 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:10.805725 3429 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:10.807813 3492 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:10.804210 3587 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:10.803607 3442 consensus_queue.cc:1035] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.068s
I20260805 22:25:10.872216 31500 raft_consensus.cc:1261] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:25:10.872841 3231 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:10.860702 3446 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:10.874476 3605 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:10.805737 3582 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:10.881246 3602 log.cc:924] Time spent T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.149s user 0.001s sys 0.000s
I20260805 22:25:10.798208 3560 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.909037 3506 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:10.909686 3506 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:10.911783 3506 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:10.891341 3587 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:10.925493 3446 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:10.928464 31706 raft_consensus.cc:1261] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:25:10.931226 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:10.932011 31705 raft_consensus.cc:2379] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 8: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:10.946449 31714 raft_consensus.cc:1226] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 8. Current term is 9. Ops: [8.5-8.5]
I20260805 22:25:10.977932 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:10.978754 31709 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:25:10.912062 3232 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:10.971228 2956 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:10.996497 31554 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:10.997743 32529 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:10.999064 31502 raft_consensus.cc:2354] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf for earlier term 8. Current term is 9.
I20260805 22:25:11.000478 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:11.002907 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:11.003574 31602 raft_consensus.cc:2454] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:25:11.003470 3560 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:11.003625 3605 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:11.007504 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:10.983178 3442 consensus_queue.cc:1035] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.016s
I20260805 22:25:11.018013 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:11.019411 32529 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:10.998764 3497 raft_consensus.cc:2781] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:11.026238 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:10.996114 2956 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:11.022414 3497 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:11.031227 2956 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:11.031395 3497 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:11.022606 3345 raft_consensus.cc:3032] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 LEADER]: Stepping down as leader of term 10
I20260805 22:25:11.033828 3345 raft_consensus.cc:729] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:11.034842 3345 consensus_queue.cc:260] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:11.036170 3345 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:11.040320 3605 raft_consensus.cc:2781] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:25:11.040833 3605 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:11.041234 3605 raft_consensus.cc:3037] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:11.045003 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:11.045722 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:11.046317 31713 raft_consensus.cc:3037] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:11.046241 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:11.046924 31594 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:25:11.047884 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:11.024103 3443 consensus_meta.cc:287] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.530s user 0.011s sys 0.000s
I20260805 22:25:11.010823 3493 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:11.024459 3527 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:11.052807 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 338927us
I20260805 22:25:11.053684 2855 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:11.054263 2855 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:11.055416 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:11.059973 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:11.061764 31554 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:11.063530 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:11.065512 31594 raft_consensus.cc:2454] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:25:11.004558 3620 raft_consensus.cc:2726] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:25:11.071030 31704 raft_consensus.cc:1261] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260805 22:25:11.075624 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:11.071820 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:11.076756 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:11.072014 31667 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:11.073798 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:11.074815 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:11.077605 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:11.078740 31604 raft_consensus.cc:2396] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for term 6 because replica has last-logged OpId of term: 4 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 2 index: 5.
I20260805 22:25:11.079385 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:11.077181 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:11.081249 31594 raft_consensus.cc:2396] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260805 22:25:11.082161 31667 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:11.050485 3232 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:11.056360 2855 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:11.090106 31706 raft_consensus.cc:1226] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5. Ops: []
I20260805 22:25:11.088310 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
I20260805 22:25:11.092514 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:11.083256 3601 consensus_queue.cc:1035] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260805 22:25:11.098798 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:11.071396 3321 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader election won for term 10
I20260805 22:25:11.100347 31667 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:11.110967 31709 raft_consensus.cc:1226] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 9. Current term is 10. Ops: []
W20260805 22:25:11.148154 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 47876.5us
I20260805 22:25:11.092460 3492 raft_consensus.cc:2726] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:11.150221 3587 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:11.152514 3587 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:11.091913 2956 raft_consensus.cc:2781] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:11.157704 2956 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:11.158073 2956 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:11.163226 3321 raft_consensus.cc:686] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 10 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:11.170693 31713 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
W20260805 22:25:11.147286 3579 log.cc:924] Time spent T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260805 22:25:11.179231 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:11.180004 31602 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:11.150050 3492 raft_consensus.cc:2781] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260805 22:25:11.207353 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 137489us
I20260805 22:25:11.232831 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:11.233594 32584 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Advancing to term 9
I20260805 22:25:11.235743 32529 leader_election.cc:398] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Vote denied by peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607) with higher term. Message: Invalid argument: T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf for earlier term 8. Current term is 9.
I20260805 22:25:11.236889 32529 leader_election.cc:401] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Cancelling election due to peer responding with higher term
I20260805 22:25:11.066707 3550 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:11.244299 32529 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:11.209535 3493 raft_consensus.cc:2726] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:11.257908 31444 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:11.259792 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:11.262039 32589 raft_consensus.cc:1261] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260805 22:25:11.210029 3492 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:11.231472 3328 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:11.264834 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:11.265915 31495 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:25:11.245643 3587 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:11.272712 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:11.273736 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:11.241643 3197 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:11.276190 31493 raft_consensus.cc:2379] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:11.256306 3303 raft_consensus.cc:2781] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:25:11.252182 3429 consensus_queue.cc:1046] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 4.3, Next index: 4, Last known committed idx: 3, Time since last communication: 0.033s
I20260805 22:25:11.291596 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:11.270131 3492 raft_consensus.cc:3037] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:11.295941 3232 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:11.395629 3569 consensus_queue.cc:1046] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 9.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.132s
I20260805 22:25:11.296933 3493 consensus_queue.cc:1035] T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.106s
I20260805 22:25:11.276216 3328 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:11.402870 3632 log.cc:924] Time spent T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.170s user 0.001s sys 0.000s
I20260805 22:25:11.392391 2855 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:25:11.404404 3631 log.cc:924] Time spent T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.081s user 0.000s sys 0.000s
I20260805 22:25:11.395817 3585 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
W20260805 22:25:11.409526 3636 log.cc:924] Time spent T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260805 22:25:11.288957 3568 consensus_queue.cc:1046] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 6.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.171s
I20260805 22:25:11.290217 3231 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:11.284186 3235 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:11.393656 3591 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:11.295887 3197 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:11.460422 3591 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:11.460328 3235 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:11.426894 31602 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 9.
I20260805 22:25:11.369299 3506 consensus_queue.cc:1035] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.041s
I20260805 22:25:11.462455 3197 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:11.333240 3549 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:11.463038 3549 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:11.274912 3443 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:11.291085 3587 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:11.464150 3506 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:11.464615 3506 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:11.462782 3235 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:11.458113 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:11.468166 3443 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:11.351624 3303 raft_consensus.cc:686] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:11.482807 3303 consensus_queue.cc:237] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:11.473668 3572 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:11.417873 3232 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:11.460191 3231 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:11.331483 3321 consensus_queue.cc:237] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:12.563181 3644 log.cc:924] Time spent T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.992s user 0.000s sys 0.001s
W20260805 22:25:11.941481 31554 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.574s user 0.005s sys 0.027s
I20260805 22:25:12.579759 3197 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:12.571465 3549 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:11.604029 3550 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:11.538825 3493 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.516093 3591 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:11.918327 3605 consensus_meta.cc:287] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.871s user 0.010s sys 0.000s
I20260805 22:25:12.581974 3591 raft_consensus.cc:3032] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 8 LEADER]: Stepping down as leader of term 8
I20260805 22:25:12.582505 3591 raft_consensus.cc:729] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
W20260805 22:25:12.544173 3500 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.940s user 0.003s sys 0.060s
W20260805 22:25:12.583235 3500 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.011s user 0.004s sys 0.061s
I20260805 22:25:12.583520 3591 consensus_queue.cc:260] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.584259 3500 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:11.539736 3638 raft_consensus.cc:2726] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:11.517118 31600 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:11.485770 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:12.586545 31600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59030 (request call id 1395) took 1129 ms. Trace:
W20260805 22:25:12.543200 31561 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.793s user 0.001s sys 0.008s
W20260805 22:25:12.587056 31561 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.837s user 0.002s sys 0.008s
W20260805 22:25:12.587340 31556 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.843s user 0.002s sys 0.023s
W20260805 22:25:12.587724 31556 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.843s user 0.002s sys 0.023s
I20260805 22:25:12.587788 3493 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.005350 3328 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:12.550518 32529 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.748s user 0.002s sys 0.006s
I20260805 22:25:11.503785 3443 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:11.623220 3497 consensus_meta.cc:287] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.591s user 0.005s sys 0.005s
W20260805 22:25:12.065001 32584 consensus_meta.cc:287] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.817s user 0.010s sys 0.000s
W20260805 22:25:12.580163 31554 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.213s user 0.015s sys 0.061s
I20260805 22:25:12.591066 32584 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 9.
W20260805 22:25:12.591248 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 1.23139e+06us
I20260805 22:25:12.591681 3620 raft_consensus.cc:2726] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: Vote denied by peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607) with higher term. Message: Invalid argument: T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf for earlier term 8. Current term is 9.
I20260805 22:25:11.793407 31499 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40740 (request call id 1404) took 1343 ms. Trace:
W20260805 22:25:12.047610 2956 consensus_meta.cc:287] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.889s user 0.011s sys 0.000s
I20260805 22:25:12.578662 31597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1397) took 1108 ms. Trace:
I20260805 22:25:11.940737 3601 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
W20260805 22:25:11.629279 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 371419us
W20260805 22:25:12.553504 31559 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 1.012s user 0.000s sys 0.009s
W20260805 22:25:12.594465 31559 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 1.084s user 0.000s sys 0.010s
W20260805 22:25:12.495371 3646 log.cc:924] Time spent T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.991s user 0.000s sys 0.001s
I20260805 22:25:12.596572 31711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 2002) took 1526 ms. Trace:
I20260805 22:25:12.596939 31711 rpcz_store.cc:276] 0805 22:25:11.069965 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:11.092366 (+ 22401us) service_pool.cc:227] Handling call
0805 22:25:11.145494 (+ 53128us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:11.235893 (+ 90399us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:11.235923 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:11.245367 (+ 9444us) log.cc:842] Serialized 26 byte log entry
0805 22:25:11.367459 (+122092us) raft_consensus.cc:1631] Marking committed up to 3
0805 22:25:11.367497 (+ 38us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:11.367562 (+ 65us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:12.596343 (+1228781us) raft_consensus.cc:1671] finished
0805 22:25:12.596360 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:12.596497 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":21717,"raft.queue_time_us":33670,"raft.run_cpu_time_us":139,"raft.run_wall_time_us":23854,"thread_start_us":116001,"threads_started":1,"wal-append.queue_time_us":176827}
I20260805 22:25:12.598418 3605 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:12.543798 3569 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.000s sys 0.007s
W20260805 22:25:12.599807 3569 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.682s user 0.000s sys 0.008s
I20260805 22:25:12.600517 3605 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:12.563042 3492 consensus_meta.cc:287] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 1.145s user 0.003s sys 0.007s
I20260805 22:25:12.582795 3231 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:12.584849 3500 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:12.604012 32529 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.804s user 0.003s sys 0.006s
W20260805 22:25:12.437196 3345 consensus_meta.cc:287] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 1.401s user 0.012s sys 0.002s
I20260805 22:25:12.606109 3232 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:12.606287 3601 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:12.600669 31450 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.641s user 0.007s sys 0.002s
W20260805 22:25:12.607348 31450 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.657s user 0.008s sys 0.002s
I20260805 22:25:11.487654 3429 raft_consensus.cc:3032] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:25:12.608088 3429 raft_consensus.cc:729] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:12.608647 3601 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:12.608907 3429 consensus_queue.cc:260] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.610100 3429 raft_consensus.cc:3037] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:12.603382 3492 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.593995 31597 rpcz_store.cc:276] 0805 22:25:11.470618 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:11.474115 (+ 3497us) service_pool.cc:227] Handling call
0805 22:25:12.578164 (+1104049us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:12.578337 (+ 173us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:12.578358 (+ 21us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:12.578393 (+ 35us) raft_consensus.cc:1631] Marking committed up to 7
0805 22:25:12.578420 (+ 27us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:12.578471 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:12.578596 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:12.610690 32581 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
W20260805 22:25:11.935118 31704 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.672s user 0.007s sys 0.018s
W20260805 22:25:12.613435 31704 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.351s user 0.015s sys 0.028s
I20260805 22:25:11.453071 3585 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.623786 31608 raft_consensus.cc:1261] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 5 index: 8. Preceding OpId from leader: term: 8 index: 10. (index mismatch)
I20260805 22:25:12.622939 3361 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.625196 3361 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.625720 3550 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:12.626312 3550 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:11.433986 2855 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.619612 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:12.622852 3492 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:12.593231 2956 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.627266 3361 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:12.627010 3585 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:12.592541 32584 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57382 (request call id 854) took 1371 ms. Trace:
I20260805 22:25:12.586987 31600 rpcz_store.cc:276] 0805 22:25:11.457619 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:11.457786 (+ 167us) service_pool.cc:227] Handling call
0805 22:25:12.586450 (+1128664us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:12.589254 3328 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
W20260805 22:25:12.494779 3568 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.686s user 0.011s sys 0.008s
W20260805 22:25:12.672612 3568 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.869s user 0.011s sys 0.009s
I20260805 22:25:12.592428 31499 rpcz_store.cc:276] 0805 22:25:10.443510 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:10.443683 (+ 173us) service_pool.cc:227] Handling call
0805 22:25:10.443959 (+ 276us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:11.046848 (+602889us) spinlock_profiling.cc:243] Waited 20.2 ms on lock 0x7b18003ec920. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f2730cf782c 00007f272e1229dc 00007f2736f59a39 00007f272e13f8bb 00007f272e13dcbf 00007f272e13d155 00007f27310603d0 00007f2731060f6b 00007f273106b78b
0805 22:25:11.046992 (+ 144us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:11.047014 (+ 22us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:11.047438 (+ 424us) log.cc:842] Serialized 26 byte log entry
0805 22:25:11.091181 (+ 43743us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:25:11.091221 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:11.091299 (+ 78us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:11.637038 (+545739us) raft_consensus.cc:1671] finished
0805 22:25:11.637062 (+ 24us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:11.787132 (+150070us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":20536,"raft.queue_time_us":43799,"raft.run_cpu_time_us":158,"raft.run_wall_time_us":7224,"spinlock_wait_cycles":46462848,"thread_start_us":88130,"threads_started":2,"wal-append.queue_time_us":203968}
I20260805 22:25:12.579429 3572 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.589967 3493 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:12.666320 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:12.669085 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
W20260805 22:25:12.670432 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 78382.7us
I20260805 22:25:12.670243 31704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58348 (request call id 635) took 1602 ms. Trace:
I20260805 22:25:12.644325 31597 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:12.671609 32584 rpcz_store.cc:276] 0805 22:25:11.221366 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:11.222916 (+ 1550us) service_pool.cc:227] Handling call
0805 22:25:12.592448 (+1369532us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:12.629709 3577 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:12.678678 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:12.663741 3321 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:12.636644 3500 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:12.680452 31486 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 1.263s user 0.000s sys 0.013s
I20260805 22:25:12.662890 3505 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.678850 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:12.683957 31594 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:25:12.668035 3601 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
W20260805 22:25:12.659580 3303 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.856s user 0.001s sys 0.004s
I20260805 22:25:12.662638 3587 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:12.693132 31486 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 1.276s user 0.000s sys 0.014s
I20260805 22:25:12.689670 31602 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 9.
I20260805 22:25:12.674034 3506 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:12.580488 3197 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:12.681872 32531 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.890s user 0.000s sys 0.023s
I20260805 22:25:12.665735 2452 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:12.590464 3497 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.676098 3585 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.665884 3232 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:12.675891 3667 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:25:12.767616 32531 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.976s user 0.000s sys 0.023s
I20260805 22:25:12.780334 31604 raft_consensus.cc:2454] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:25:12.682868 31704 rpcz_store.cc:276] 0805 22:25:11.067863 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:11.080942 (+ 13079us) service_pool.cc:227] Handling call
0805 22:25:11.084405 (+ 3463us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:11.084591 (+ 186us) raft_consensus.cc:1520] Early marking committed up to index 8
0805 22:25:11.262473 (+177882us) log.cc:842] Serialized 14 byte log entry
0805 22:25:12.669925 (+1407452us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:12.669975 (+ 50us) raft_consensus.cc:1631] Marking committed up to 8
0805 22:25:12.670015 (+ 40us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:12.670075 (+ 60us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:12.670173 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":108847,"prepare.queue_time_us":179687,"prepare.run_cpu_time_us":31,"prepare.run_wall_time_us":32,"raft.queue_time_us":69163,"spinlock_wait_cycles":73472,"thread_start_us":1448727,"threads_started":2,"wal-append.queue_time_us":1412257}
I20260805 22:25:12.760241 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:12.763393 32532 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.889s user 0.015s sys 0.028s
W20260805 22:25:12.765650 3303 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.967s user 0.001s sys 0.005s
I20260805 22:25:12.678730 3345 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:12.681679 3050 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.679158 3650 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:12.700491 3605 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
W20260805 22:25:12.783903 32532 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.910s user 0.015s sys 0.028s
I20260805 22:25:12.784771 31596 raft_consensus.cc:1226] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 6. Current term is 7. Ops: []
I20260805 22:25:12.783313 31603 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:25:12.767242 3500 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:12.770641 3547 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.677085 3328 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.786756 31597 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 9.
I20260805 22:25:12.771809 3662 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.773346 3506 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.794422 31486 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40740 (request call id 1413) took 1532 ms. Trace:
I20260805 22:25:12.800097 3491 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.741077 3321 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.768806 3601 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.778628 3585 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.766594 3505 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.773368 2452 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.801515 3547 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.807746 3655 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:12.793107 2956 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:12.815747 31486 rpcz_store.cc:276] 0805 22:25:11.261534 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:11.262050 (+ 516us) service_pool.cc:227] Handling call
0805 22:25:11.283242 (+ 21192us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:11.353149 (+ 69907us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:11.353184 (+ 35us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:11.377075 (+ 23891us) log.cc:842] Serialized 26 byte log entry
0805 22:25:12.764003 (+1386928us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:25:12.764039 (+ 36us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:12.764110 (+ 71us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:12.794222 (+ 30112us) raft_consensus.cc:1671] finished
0805 22:25:12.794240 (+ 18us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:12.794364 (+ 124us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12737,"raft.queue_time_us":83519,"raft.run_cpu_time_us":120,"raft.run_wall_time_us":3694,"thread_start_us":1346927,"threads_started":1,"wal-append.queue_time_us":1398628}
I20260805 22:25:12.793747 3345 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.799817 2855 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:25:12.798050 3497 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:12.798463 3490 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:12.824532 3491 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.789595 3667 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.825464 32589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57366 (request call id 656) took 1361 ms. Trace:
I20260805 22:25:12.825448 3651 consensus_queue.cc:1035] T 6b16a2e0814140e39609da87479bb57e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:25:12.827957 3568 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.798672 3687 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.816749 3686 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:12.786762 3232 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.868196 3345 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:12.816080 3666 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:12.869299 3321 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:12.869454 3572 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:12.868948 3666 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.869980 3572 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:12.870443 3572 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.870815 3547 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:12.699352 3577 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.796437 3050 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.871666 3491 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:12.872400 3572 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:12.872730 3654 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.872929 3572 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
W20260805 22:25:12.837260 31448 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.697s user 0.002s sys 0.011s
I20260805 22:25:12.873224 3577 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:12.868233 3678 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:12.873883 3687 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.874224 3568 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.875362 3232 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:12.847522 3661 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:12.873272 3654 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.876487 3661 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.878319 3654 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:12.878754 3620 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:12.873818 3577 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.876109 3568 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:12.856721 3490 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.880326 3577 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:12.873576 31448 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.890s user 0.002s sys 0.012s
I20260805 22:25:12.871606 3666 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:12.882246 3666 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:12.882776 3666 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.867105 3686 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.883178 3678 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.883726 3676 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:12.884935 3686 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:12.799688 3500 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.835083 3672 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.816185 3662 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.886539 3505 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:12.856931 32589 rpcz_store.cc:276] 0805 22:25:11.464433 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:11.464748 (+ 315us) service_pool.cc:227] Handling call
0805 22:25:11.465109 (+ 361us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:11.465466 (+ 357us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:11.465498 (+ 32us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:11.465943 (+ 445us) log.cc:842] Serialized 26 byte log entry
0805 22:25:11.932907 (+466964us) raft_consensus.cc:1631] Marking committed up to 7
0805 22:25:11.932950 (+ 43us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:11.933084 (+ 134us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:12.825204 (+892120us) raft_consensus.cc:1671] finished
0805 22:25:12.825225 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:12.825357 (+ 132us) inbound_call.cc:173] Queueing success response
Metrics: {"raft.queue_time_us":1195,"raft.run_cpu_time_us":113,"raft.run_wall_time_us":114,"thread_start_us":466601,"threads_started":1,"wal-append.queue_time_us":1323318}
W20260805 22:25:12.862035 3659 log.cc:924] Time spent T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.070s user 0.000s sys 0.001s
I20260805 22:25:12.863309 3585 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:12.865259 3601 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:12.850472 3655 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.796919 3650 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.857088 3497 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.884896 3676 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.885555 3577 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:12.886330 3666 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:12.882898 3050 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:12.881587 3675 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:12.791226 3664 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:12.887207 3321 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:12.900665 31609 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 10.
I20260805 22:25:12.881309 3490 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:12.885183 3678 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:12.908409 3197 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:12.897754 3572 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.799185 3605 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.856906 3550 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:12.906205 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:12.876014 3687 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:12.934173 31715 raft_consensus.cc:2454] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:25:12.909665 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:12.911603 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:12.912343 3585 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.912781 31444 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:12.914361 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:12.915766 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:12.879082 3661 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:12.903018 3491 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.898414 3232 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:12.919173 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:12.921520 31667 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:12.857805 3506 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.924044 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:12.906929 3686 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.904223 2855 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:12.906418 3547 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:12.903400 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:12.924083 3328 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:12.908516 3345 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:12.948945 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:12.928119 3577 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:12.879264 3620 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.936626 31601 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1390) took 1875 ms. Trace:
I20260805 22:25:12.950630 32579 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:25:12.951486 31707 raft_consensus.cc:2379] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 10: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:12.951561 31709 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
I20260805 22:25:12.923715 3429 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:12.919703 3601 raft_consensus.cc:482] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:12.956017 31444 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:12.934836 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 252107us
I20260805 22:25:12.958585 31667 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:12.967374 32591 raft_consensus.cc:2396] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 9 because replica has last-logged OpId of term: 8 index: 10, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 8.
I20260805 22:25:12.949401 32584 raft_consensus.cc:2379] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 7: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:12.968820 31704 raft_consensus.cc:3037] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:12.931290 3666 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:12.922403 3497 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.934782 3664 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.934487 3321 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.904474 3672 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:13.043363 32581 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57372 (request call id 726) took 1559 ms. Trace:
I20260805 22:25:13.039872 2762 raft_consensus.cc:2781] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election won for term 7
W20260805 22:25:13.099761 3699 log.cc:924] Time spent T 0191f03fa8984a40afc7ec1e44970c8e P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.075s user 0.000s sys 0.000s
I20260805 22:25:12.966910 2855 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:13.098348 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:13.101198 31715 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 9.
I20260805 22:25:13.101456 3572 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:13.102311 3700 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:13.102437 2855 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:13.097219 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:12.952771 3490 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:13.103914 31708 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 11.
I20260805 22:25:13.104247 3490 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:13.102875 3700 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.958411 3550 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:13.091212 31706 raft_consensus.cc:1226] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5. Ops: []
I20260805 22:25:13.106565 3700 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:13.107079 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 148569us
I20260805 22:25:13.109157 31667 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:13.062995 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:13.110131 32584 raft_consensus.cc:2454] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:25:12.954363 3585 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.935026 3675 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:13.115981 3585 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:12.968192 3547 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:13.116779 3675 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:13.118068 3711 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:13.118872 3711 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:13.120903 3650 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:12.953193 3491 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:12.956323 3232 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:13.122179 3711 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:13.122803 3664 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:13.123208 3232 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:13.123296 3700 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:13.008502 3328 raft_consensus.cc:2726] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:13.123623 3708 raft_consensus.cc:2781] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:25:13.124285 3708 raft_consensus.cc:686] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:13.125001 3550 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:13.125609 3497 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:13.125265 3708 consensus_queue.cc:237] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 4, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:12.967285 3686 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:13.126011 2762 raft_consensus.cc:686] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [term 7 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:13.126658 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:13.121577 3620 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:13.126991 2762 consensus_queue.cc:237] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:13.128336 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:13.128925 32583 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:25:13.024721 3443 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:13.129946 3686 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:13.130620 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:13.132213 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:13.135003 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:13.109474 32581 rpcz_store.cc:276] 0805 22:25:11.484214 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:11.485469 (+ 1255us) service_pool.cc:227] Handling call
0805 22:25:13.043163 (+1557694us) spinlock_profiling.cc:243] Waited 23.7 ms on lock 0x7b54004303d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:13.043294 (+ 131us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":54518016}
I20260805 22:25:13.123935 3700 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:13.130362 3547 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:13.034988 3655 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:13.130055 3443 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:13.188980 3443 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:13.098295 3506 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:13.041889 3605 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:13.010808 3696 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:13.032156 3667 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:13.189456 3700 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:12.986583 3429 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:13.030797 3668 log.cc:924] Time spent T f8863c96c162438983f403d38b11c5db P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.053s user 0.003s sys 0.000s
I20260805 22:25:12.967515 3197 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:13.034224 3707 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:13.032490 3692 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:13.122701 3711 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:13.224071 3711 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:13.167616 31501 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40740 (request call id 1407) took 1910 ms. Trace:
I20260805 22:25:13.189877 3696 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:13.056375 2452 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:13.030961 3705 raft_consensus.cc:482] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:13.109910 2855 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:13.040144 3697 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:12.969452 31601 rpcz_store.cc:276] 0805 22:25:11.060742 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:11.060861 (+ 119us) service_pool.cc:227] Handling call
0805 22:25:11.072720 (+ 11859us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:11.207913 (+135193us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:11.207941 (+ 28us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:11.231660 (+ 23719us) log.cc:842] Serialized 26 byte log entry
0805 22:25:11.457292 (+225632us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:25:11.457329 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:11.457930 (+ 601us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:12.936408 (+1478478us) raft_consensus.cc:1671] finished
0805 22:25:12.936429 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:12.936565 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":172218,"raft.queue_time_us":136484,"raft.run_cpu_time_us":134,"raft.run_wall_time_us":10132,"thread_start_us":196674,"threads_started":1,"wal-append.queue_time_us":1639653}
I20260805 22:25:13.136366 3490 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:14.597841 29789 debug-util.cc:398] Leaking SignalData structure 0x7b080000ee80 after lost signal to thread 31668
W20260805 22:25:14.598326 29789 kernel_stack_watchdog.cc:198] Thread 31668 stuck at /data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/rpc/outbound_call.cc:308 for 398ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
I20260805 22:25:13.135660 3321 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:13.134549 3491 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:13.194197 3429 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:13.030696 3500 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:13.137508 3550 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:13.041275 3591 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:13.010849 3601 raft_consensus.cc:504] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:13.030061 3653 log.cc:924] Time spent T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.070s user 0.001s sys 0.002s
I20260805 22:25:13.038368 3662 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:13.134112 3672 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:13.037021 3676 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:13.075331 3666 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:14.655535 31357 catalog_manager.cc:5495] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 5 to 8. New cstate: current_term: 8 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:25:14.657301 3666 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:13.202461 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 246504us
I20260805 22:25:14.657091 31357 catalog_manager.cc:5495] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 5 to 8, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 8 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:25:14.658260 31444 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:13.223811 3707 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:13.224927 31501 rpcz_store.cc:276] 0805 22:25:11.257185 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:11.257285 (+ 100us) service_pool.cc:227] Handling call
0805 22:25:13.166815 (+1909530us) spinlock_profiling.cc:243] Waited 1.13 ms on lock 0x7b5800001898. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:25:13.167062 (+ 247us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:13.167271 (+ 209us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:13.167295 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:13.167337 (+ 42us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:25:13.167372 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:13.167426 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:13.167548 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2603392}
I20260805 22:25:14.661974 3235 raft_consensus.cc:2726] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:25:13.226433 3711 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:13.132777 32594 raft_consensus.cc:2454] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:25:13.227027 3696 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:13.223837 3692 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:13.353907 31704 raft_consensus.cc:2454] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 9.
I20260805 22:25:13.431285 3690 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:13.636034 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:25:13.647962 3634 log.cc:924] Time spent T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 2.497s user 0.008s sys 0.022s
I20260805 22:25:12.971285 3345 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:13.810158 3705 raft_consensus.cc:504] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:13.809787 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:25:13.818564 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 92009.2us
W20260805 22:25:14.199218 31398 debug-util.cc:398] Leaking SignalData structure 0x7b080003ab00 after lost signal to thread 29783
I20260805 22:25:14.394806 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:14.396677 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:14.404297 3572 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:13.135339 32583 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:25:13.010108 3577 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:14.506040 3697 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:14.505990 2855 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:14.507663 31596 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56384 (request call id 439) took 3507 ms. Trace:
I20260805 22:25:14.508306 3496 raft_consensus.cc:3032] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 9 LEADER]: Stepping down as leader of term 9
I20260805 22:25:14.512602 3605 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:13.029893 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:14.551313 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:14.553751 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:14.554534 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:25:14.555294 31492 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:40740 (request call id 1415) took 3280 ms (client timeout 2048 ms). Trace:
I20260805 22:25:14.553145 31499 raft_consensus.cc:1261] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260805 22:25:14.556627 31490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40740 (request call id 1411) took 3298 ms. Trace:
I20260805 22:25:14.557243 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:25:14.559852 31609 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:59028 (request call id 2461) took 1895 ms (client timeout 1872 ms). Trace:
I20260805 22:25:14.562435 31711 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 2408) took 1660 ms. Trace:
I20260805 22:25:14.564142 31447 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
W20260805 22:25:14.577785 32529 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.579s (SENT) [suppressed 7 similar messages]
I20260805 22:25:14.582275 3701 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:14.586866 3297 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:14.586854 3527 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:14.584327 3655 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:14.610011 3650 raft_consensus.cc:482] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:14.607765 3651 raft_consensus.cc:482] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:14.607373 3499 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:14.611326 3682 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:14.611735 3664 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:14.612128 3667 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:14.589217 3549 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:14.612524 3568 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:14.606454 3493 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:14.611055 3443 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:14.627477 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:14.627766 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:14.628326 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:14.628458 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:14.628638 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:14.628857 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:14.629779 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:14.629897 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:14.631012 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:14.631582 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:14.632378 3601 leader_election.cc:288] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:14.626289 3500 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
W20260805 22:25:14.634382 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 1.52517e+06us
I20260805 22:25:14.634907 3708 raft_consensus.cc:2781] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:14.638237 32579 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57366 (request call id 665) took 1536 ms. Trace:
I20260805 22:25:14.643013 31554 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:14.644019 31709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58348 (request call id 637) took 1566 ms. Trace:
I20260805 22:25:14.646631 31713 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58348 (request call id 636) took 3381 ms. Trace:
I20260805 22:25:11.162853 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:25:14.653234 3684 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:14.659840 31444 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:14.892027 3281 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:14.893200 3281 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:14.895607 3281 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:14.897562 31708 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 2013) took 1566 ms (client timeout 1703 ms). Trace:
I20260805 22:25:14.660115 31357 catalog_manager.cc:5495] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 6 to 7, leader changed from 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 7 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
W20260805 22:25:14.898031 31708 rpcz_store.cc:269] 0805 22:25:13.330734 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:14.396382 (+1065648us) service_pool.cc:227] Handling call
0805 22:25:14.897493 (+501111us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:14.900629 31597 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 7.
I20260805 22:25:14.897006 31594 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
W20260805 22:25:14.898896 31398 debug-util.cc:398] Leaking SignalData structure 0x7b0800033580 after lost signal to thread 29786
I20260805 22:25:14.700431 32587 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57366 (request call id 666) took 1557 ms. Trace:
W20260805 22:25:14.902753 31398 debug-util.cc:398] Leaking SignalData structure 0x7b08002a6260 after lost signal to thread 31402
I20260805 22:25:14.900496 31711 rpcz_store.cc:276] 0805 22:25:12.902256 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:12.902398 (+ 142us) service_pool.cc:227] Handling call
0805 22:25:12.908376 (+ 5978us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:13.101305 (+192929us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:13.101334 (+ 29us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:13.101393 (+ 59us) raft_consensus.cc:1631] Marking committed up to 8
0805 22:25:13.101425 (+ 32us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:14.562225 (+1460800us) spinlock_profiling.cc:243] Waited 193 ms on lock 0x7b5400033ed4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:14.562266 (+ 41us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:14.562380 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":443249536}
I20260805 22:25:14.904019 3281 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader election won for term 9
I20260805 22:25:14.904721 3281 raft_consensus.cc:686] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 9 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:14.905649 3281 consensus_queue.cc:237] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:14.906629 3297 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:14.906821 3664 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:14.907953 3707 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:14.908135 3655 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:14.908743 32585 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57372 (request call id 729) took 1956 ms (client timeout 2155 ms). Trace:
I20260805 22:25:14.909369 3605 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:14.909557 3684 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:14.910295 3655 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:14.911047 31715 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 2009) took 1581 ms (client timeout 1867 ms). Trace:
I20260805 22:25:14.900013 31486 raft_consensus.cc:2396] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 10 because replica has last-logged OpId of term: 8 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 4.
I20260805 22:25:14.899224 3496 raft_consensus.cc:729] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 9 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:14.664428 3197 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:14.912719 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:14.670653 3492 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:14.914048 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:14.913910 3492 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:14.911499 31715 rpcz_store.cc:269] 0805 22:25:13.329663 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:13.635665 (+306002us) service_pool.cc:227] Handling call
0805 22:25:14.910963 (+1275298us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:14.914867 3650 raft_consensus.cc:504] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:14.915717 3297 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:14.906124 3345 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:14.913393 31486 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 8.
I20260805 22:25:14.916801 3650 leader_election.cc:288] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:14.917332 3651 raft_consensus.cc:504] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:14.903121 31398 debug-util.cc:398] Leaking SignalData structure 0x7b08003d0a00 after lost signal to thread 31423
I20260805 22:25:14.918226 3682 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:14.893105 3500 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:14.919337 31398 debug-util.cc:398] Leaking SignalData structure 0x7b0800050ba0 after lost signal to thread 31535
I20260805 22:25:14.916575 31706 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 2010) took 1570 ms. Trace:
W20260805 22:25:14.919652 31398 debug-util.cc:398] Leaking SignalData structure 0x7b080032f080 after lost signal to thread 31536
I20260805 22:25:14.913271 3197 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:14.921440 3549 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:14.922744 31398 debug-util.cc:398] Leaking SignalData structure 0x7b080032a800 after lost signal to thread 31643
W20260805 22:25:14.923022 31398 debug-util.cc:398] Leaking SignalData structure 0x7b080019d060 after lost signal to thread 31646
I20260805 22:25:14.923347 3197 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:14.923712 3549 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:14.924067 31398 debug-util.cc:398] Leaking SignalData structure 0x7b080039bfc0 after lost signal to thread 31748
I20260805 22:25:14.924355 31493 raft_consensus.cc:2379] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:14.924461 31597 raft_consensus.cc:2454] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:25:14.924208 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:14.921358 3297 raft_consensus.cc:2726] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:14.914749 3568 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:14.926928 3655 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:25:14.920302 32585 rpcz_store.cc:269] 0805 22:25:12.952860 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:12.953001 (+ 141us) service_pool.cc:227] Handling call
0805 22:25:14.908675 (+1955674us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:14.927901 3568 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:14.928222 31502 raft_consensus.cc:2379] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:14.906518 3697 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:14.920249 3496 consensus_queue.cc:260] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:14.920903 31706 rpcz_store.cc:276] 0805 22:25:13.330004 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:13.809473 (+479469us) service_pool.cc:227] Handling call
0805 22:25:14.900716 (+1091243us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:14.925560 3690 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:14.911953 3692 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:14.902408 32587 rpcz_store.cc:276] 0805 22:25:13.102543 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:13.102756 (+ 213us) service_pool.cc:227] Handling call
0805 22:25:13.103111 (+ 355us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:14.658904 (+1555793us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:14.658934 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:14.658971 (+ 37us) raft_consensus.cc:1631] Marking committed up to 7
0805 22:25:14.659000 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:14.659469 (+ 469us) spinlock_profiling.cc:243] Waited 1.56 s on lock 0x7b54005d52d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:14.659505 (+ 36us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:14.659623 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3577538432}
I20260805 22:25:14.928767 3655 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:14.920239 3682 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:14.920624 3499 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:14.944893 31705 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56260 (request call id 2409) took 2042 ms. Trace:
I20260805 22:25:14.926939 3500 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:14.896953 31500 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:25:14.900597 31604 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 8.
W20260805 22:25:14.901566 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 241799us
I20260805 22:25:14.945281 31705 rpcz_store.cc:276] 0805 22:25:12.902607 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:12.902940 (+ 333us) service_pool.cc:227] Handling call
0805 22:25:12.909905 (+ 6965us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:14.904730 (+1994825us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:14.904763 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:14.904802 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:25:14.904831 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:14.944681 (+ 39850us) spinlock_profiling.cc:243] Waited 1.99 s on lock 0x7b5400219654. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:14.944725 (+ 44us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:14.944834 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4587198848}
I20260805 22:25:14.947556 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:14.926149 3708 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:14.900331 32583 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57372 (request call id 736) took 1772 ms. Trace:
I20260805 22:25:14.920434 3701 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:14.945793 3505 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:14.900086 31713 rpcz_store.cc:276] 0805 22:25:11.265347 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:11.273831 (+ 8484us) service_pool.cc:227] Handling call
0805 22:25:11.274164 (+ 333us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:11.285033 (+ 10869us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:11.285066 (+ 33us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:11.301686 (+ 16620us) log.cc:842] Serialized 26 byte log entry
0805 22:25:12.606977 (+1305291us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:25:12.607018 (+ 41us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:12.607156 (+ 138us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:13.844626 (+1237470us) spinlock_profiling.cc:243] Waited 18.6 ms on lock 0x7b4800020358. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2735f0f7f1 00007f2735f068e7 00007f2735f0677a 00007f2735ffa95f 00007f2735ffa627 00007f273600eb2c 00007f273600ead8 00007f273600ea58 00007f273600ea13 00007f273600dcbc 00007f2734f8fbe6
0805 22:25:14.646309 (+801683us) raft_consensus.cc:1671] finished
0805 22:25:14.646334 (+ 25us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:14.646554 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":831796,"raft.queue_time_us":15223,"raft.run_cpu_time_us":30243,"raft.run_wall_time_us":2549436,"spinlock_wait_cycles":42702208,"thread_start_us":107283,"threads_started":1,"wal-append.queue_time_us":1423210}
W20260805 22:25:14.951436 32594 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57366 (request call id 662) took 2037 ms (client timeout 1853 ms). Trace:
I20260805 22:25:14.951515 3505 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:14.943728 3496 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:14.952872 3701 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:14.900207 31707 raft_consensus.cc:2454] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
I20260805 22:25:14.953683 31707 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 2012) took 1623 ms. Trace:
I20260805 22:25:14.954093 31707 rpcz_store.cc:276] 0805 22:25:13.330445 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:14.394473 (+1064028us) service_pool.cc:227] Handling call
0805 22:25:14.953624 (+559151us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260805 22:25:14.948711 3655 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:14.929811 31398 debug-util.cc:398] Leaking SignalData structure 0x7b08002f9aa0 after lost signal to thread 31749
W20260805 22:25:14.956236 31398 debug-util.cc:398] Leaking SignalData structure 0x7b080035d900 after lost signal to thread 32633
W20260805 22:25:14.956570 31398 debug-util.cc:398] Leaking SignalData structure 0x7b08002e4600 after lost signal to thread 32634
I20260805 22:25:14.943718 2855 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:14.958328 31600 raft_consensus.cc:2396] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 4 index: 6.
I20260805 22:25:14.958591 3651 leader_election.cc:288] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:14.959302 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:14.959738 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:14.918591 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 1.83976e+06us
I20260805 22:25:14.927090 31608 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 7.
I20260805 22:25:14.912425 3667 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:14.925226 31556 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
W20260805 22:25:14.950255 31704 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56260 (request call id 2407) took 2048 ms (client timeout 2091 ms). Trace:
W20260805 22:25:14.896416 31492 rpcz_store.cc:269] 0805 22:25:11.274513 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:14.555084 (+3280571us) service_pool.cc:214] Skipping call since client already timed out
0805 22:25:14.555207 (+ 123us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20260805 22:25:14.903358 31494 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 9.
W20260805 22:25:14.921358 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 357291us
I20260805 22:25:14.960533 31600 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 8.
I20260805 22:25:14.909042 3664 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:14.895725 3577 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:14.662791 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:14.893961 31603 raft_consensus.cc:2379] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 6: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:15.007325 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:14.926462 32579 rpcz_store.cc:276] 0805 22:25:13.102241 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:13.107050 (+ 4809us) service_pool.cc:227] Handling call
0805 22:25:13.107320 (+ 270us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:14.623289 (+1515969us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:14.623319 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:14.623355 (+ 36us) raft_consensus.cc:1631] Marking committed up to 9
0805 22:25:14.623385 (+ 30us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:14.637983 (+ 14598us) spinlock_profiling.cc:243] Waited 1.52 s on lock 0x7b5400652054. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:14.638031 (+ 48us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:14.638176 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3485923584}
I20260805 22:25:14.896473 31596 rpcz_store.cc:276] 0805 22:25:10.999917 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:11.000459 (+ 542us) service_pool.cc:227] Handling call
0805 22:25:11.010187 (+ 9728us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:12.795789 (+1785602us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:14.503612 (+1707823us) spinlock_profiling.cc:243] Waited 1.77 s on lock 0x7b54002396d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:14.507570 (+ 3958us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4080687360}
I20260805 22:25:14.917658 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:14.949298 32589 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57372 (request call id 727) took 2015 ms. Trace:
I20260805 22:25:14.934844 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:14.928611 3568 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260805 22:25:15.009444 31499 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 8.
I20260805 22:25:14.951826 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:15.011415 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.011976 31502 raft_consensus.cc:2379] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:14.952333 31602 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
I20260805 22:25:14.953563 3505 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:14.959954 31500 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 8.
I20260805 22:25:14.916047 3492 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:14.899493 31609 rpcz_store.cc:269] 0805 22:25:12.664522 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:12.664669 (+ 147us) service_pool.cc:227] Handling call
0805 22:25:14.559678 (+1895009us) spinlock_profiling.cc:243] Waited 200 ms on lock 0x7b54001441d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:14.559784 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":459773312}
I20260805 22:25:14.934713 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:25:14.926867 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 283920us
I20260805 22:25:15.012135 31600 raft_consensus.cc:2454] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:25:14.927335 3493 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:14.914443 3705 leader_election.cc:288] T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:14.926146 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:15.037330 31597 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:15.041666 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.042397 31608 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:25:14.928635 3443 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:15.043336 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:15.043761 31667 connection.cc:232] RPC call timeout handler was delayed by -2.2564988089725375s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1784s for a response
I20260805 22:25:15.044044 31608 raft_consensus.cc:2379] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 9: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:15.044486 3443 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:15.044880 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.047650 3499 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:15.049964 3690 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:14.925251 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:15.050804 31493 raft_consensus.cc:3037] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:15.051168 31486 raft_consensus.cc:1261] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20260805 22:25:15.052989 31667 connection.cc:232] RPC call timeout handler was delayed by -2.1358988089486957s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1918s for a response
W20260805 22:25:14.952032 32594 rpcz_store.cc:269] 0805 22:25:12.914028 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:12.914131 (+ 103us) service_pool.cc:227] Handling call
0805 22:25:14.951281 (+2037150us) spinlock_profiling.cc:243] Waited 185 ms on lock 0x7b54000ae454. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:14.951373 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":424418048}
I20260805 22:25:14.901827 31490 rpcz_store.cc:276] 0805 22:25:11.258346 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:11.258456 (+ 110us) service_pool.cc:227] Handling call
0805 22:25:14.555949 (+3297493us) spinlock_profiling.cc:243] Waited 1.09 ms on lock 0x7b5800001898. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108702 00007f2736ff8b09 00007f2735f34730 00007f2735ffa79b 00007f2735ffa821 00007f2735fbe2b0 00007f2735fa97f9 00007f2735fa92a2 00007f272f68f684
0805 22:25:14.556150 (+ 201us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:14.556307 (+ 157us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:14.556331 (+ 24us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:14.556370 (+ 39us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:25:14.556398 (+ 28us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:14.556449 (+ 51us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:14.556554 (+ 105us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2513280}
I20260805 22:25:14.950469 32583 rpcz_store.cc:276] 0805 22:25:13.127463 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:13.131954 (+ 4491us) service_pool.cc:227] Handling call
0805 22:25:14.900273 (+1768319us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260805 22:25:14.966406 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 388621us
W20260805 22:25:14.962924 31704 rpcz_store.cc:269] 0805 22:25:12.901812 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:12.901975 (+ 163us) service_pool.cc:227] Handling call
0805 22:25:14.949963 (+2047988us) spinlock_profiling.cc:243] Waited 9.86 ms on lock 0x7b54002b39d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:14.950071 (+ 108us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":22671616}
I20260805 22:25:14.964515 31604 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 10.
I20260805 22:25:14.911761 3605 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:15.003052 31601 raft_consensus.cc:2454] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:25:14.964881 31444 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:15.008020 31603 raft_consensus.cc:2454] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:25:15.009002 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.058917 31494 raft_consensus.cc:2454] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:25:15.044497 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.009174 32589 rpcz_store.cc:276] 0805 22:25:12.933461 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:12.933598 (+ 137us) service_pool.cc:227] Handling call
0805 22:25:12.955213 (+ 21615us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:14.905222 (+1950009us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:14.905252 (+ 30us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:14.905288 (+ 36us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:25:14.905322 (+ 34us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:14.949070 (+ 43748us) spinlock_profiling.cc:243] Waited 1.95 s on lock 0x7b54005028d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:14.949117 (+ 47us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:14.949239 (+ 122us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":4484166528}
I20260805 22:25:15.007325 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:14.899767 31709 rpcz_store.cc:276] 0805 22:25:13.077299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:13.077455 (+ 156us) service_pool.cc:227] Handling call
0805 22:25:13.103909 (+ 26454us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:14.491460 (+1387551us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:14.491491 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:14.491539 (+ 48us) raft_consensus.cc:1631] Marking committed up to 9
0805 22:25:14.491568 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:14.643778 (+152210us) spinlock_profiling.cc:243] Waited 1.39 s on lock 0x7b5400203d54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:14.643832 (+ 54us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:14.643958 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3190487808}
I20260805 22:25:15.011415 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.012988 31357 catalog_manager.cc:5495] T 6a1aa2dc5e5341f0a0450361d6c95814 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 3 to 5, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to 6bf8c694a0ba40978195ae044a917caf (127.29.21.196). New cstate: current_term: 5 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:25:15.017752 31668 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:14.927652 3572 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:15.001926 3297 consensus_queue.cc:1035] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.017s
I20260805 22:25:14.963804 3708 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:15.050030 3684 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:14.973239 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 48072us
I20260805 22:25:15.072360 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.057135 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.060962 31493 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
I20260805 22:25:15.062073 31554 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:15.062908 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.063514 31608 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:15.066108 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.067414 31597 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 9.
W20260805 22:25:15.069262 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 104461us
I20260805 22:25:15.078194 31495 raft_consensus.cc:1261] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
W20260805 22:25:15.074417 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 56748.1us
I20260805 22:25:15.080114 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.080684 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.081261 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.081913 31602 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:25:15.078310 31444 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:15.079931 3651 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260805 22:25:15.081329 31601 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:15.083832 3651 raft_consensus.cc:482] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:15.082221 31668 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:15.084872 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.085623 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.080108 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.086520 32583 raft_consensus.cc:2454] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 4.
I20260805 22:25:15.086516 31594 raft_consensus.cc:2454] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 3.
I20260805 22:25:15.010762 3568 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:14.906536 3527 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:15.088778 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.075208 31357 catalog_manager.cc:5495] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:25:15.091253 31554 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:15.093365 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.098076 32579 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:15.093055 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.099059 31604 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:25:15.097079 31357 catalog_manager.cc:5495] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 2 to 3, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:25:15.101078 31357 catalog_manager.cc:5495] T c75e772a0f924080865ddf85ed1e2d86 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:25:15.094818 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.102717 31600 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 11.
I20260805 22:25:15.105042 31357 catalog_manager.cc:5495] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 5 to 8, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 8 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:25:15.106990 31357 catalog_manager.cc:5495] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 3 to 4, leader changed from 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 4 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:25:15.108521 31357 catalog_manager.cc:5495] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 5 to 7. New cstate: current_term: 7 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20260805 22:25:15.112519 31359 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:34400 (request call id 286) took 14237 ms (client timeout 15000 ms). Trace:
I20260805 22:25:15.113559 3321 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260805 22:25:15.113996 3321 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:15.114084 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.114913 32587 raft_consensus.cc:2454] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:25:15.121927 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.110173 31357 catalog_manager.cc:5495] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 5 to 6, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195). New cstate: current_term: 6 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:25:15.095963 3690 raft_consensus.cc:2781] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:15.123909 3690 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:15.124377 3690 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:15.124241 31357 catalog_manager.cc:5495] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "6bf8c694a0ba40978195ae044a917caf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20260805 22:25:15.125422 3667 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:15.093783 2855 raft_consensus.cc:2781] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:15.126754 2855 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:15.127269 2855 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:15.125994 31357 catalog_manager.cc:5495] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:25:15.131856 31357 catalog_manager.cc:5495] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf reported cstate change: term changed from 2 to 3, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 3 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } } }
I20260805 22:25:15.084231 3651 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:15.126292 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.063146 3650 raft_consensus.cc:2781] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election won for term 6
W20260805 22:25:15.139205 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 48021.7us
I20260805 22:25:15.143406 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.144346 32583 raft_consensus.cc:2396] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for term 8 because replica has last-logged OpId of term: 7 index: 8, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 7.
I20260805 22:25:15.144723 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.114394 3321 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:15.144284 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.149708 3584 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:15.155047 32586 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:57372 (request call id 730) took 2197 ms. Trace:
I20260805 22:25:15.152925 31704 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 10.
I20260805 22:25:15.157956 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.158584 32579 raft_consensus.cc:2454] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:25:15.159477 31554 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:15.154906 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.160351 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.154963 3584 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:15.161188 31705 raft_consensus.cc:2396] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260805 22:25:15.161368 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.161543 3697 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:15.162478 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.162222 3697 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:15.164147 3572 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:15.165459 3577 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:15.165946 3577 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:15.167114 3527 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:15.138533 3505 raft_consensus.cc:2781] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:25:15.168111 3505 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:15.168443 3505 raft_consensus.cc:3037] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:15.167878 31608 raft_consensus.cc:1261] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260805 22:25:15.165393 3493 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:15.170794 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.167835 3577 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:15.171558 31704 raft_consensus.cc:2379] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 9: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:15.165189 3697 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:15.160436 32594 raft_consensus.cc:2396] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 7 because replica has last-logged OpId of term: 5 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 3 index: 5.
I20260805 22:25:15.172772 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.172842 3584 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:15.173978 3493 raft_consensus.cc:2726] T 527d60ab6db74595bed7e87ca2ecc6e8 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:15.175887 3650 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:15.141256 3667 raft_consensus.cc:2781] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:25:15.176831 3667 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:15.177170 3667 raft_consensus.cc:3037] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:15.178756 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.179389 32589 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:25:15.095783 3568 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Advancing to term 12
I20260805 22:25:15.179783 3493 consensus_queue.cc:1035] T 8abfbdc4afff482ebc81c4fda5d0812a P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20260805 22:25:15.181273 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.181922 32581 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 9.
I20260805 22:25:15.182747 31554 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:15.183202 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.183867 3493 raft_consensus.cc:2726] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:15.184211 31709 raft_consensus.cc:1226] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 9. Current term is 10. Ops: []
I20260805 22:25:15.186108 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.187696 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.188285 31609 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 7.
I20260805 22:25:15.189726 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.191090 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.191845 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.180635 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.192721 32579 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:25:15.173892 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.193692 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:15.192931 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 139383us
I20260805 22:25:15.194376 32583 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:25:15.191668 32587 raft_consensus.cc:2454] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:25:15.193622 32579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.196107 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.198436 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:15.194447 31667 connection.cc:232] RPC call timeout handler was delayed by -2.018898808863014s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2048s for a response
W20260805 22:25:15.204689 31667 connection.cc:232] RPC call timeout handler was delayed by -1.9675988089293242s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.21050000000000002s for a response
W20260805 22:25:15.205502 31667 connection.cc:232] RPC call timeout handler was delayed by -1.9031219899188727s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1698s for a response
I20260805 22:25:15.206555 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.207342 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.209733 32591 raft_consensus.cc:2454] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 3.
W20260805 22:25:15.210496 31667 connection.cc:232] RPC call timeout handler was delayed by -1.4689988088794053s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.26589999999999997s for a response
W20260805 22:25:15.112941 31359 rpcz_store.cc:269] 0805 22:25:00.875025 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:00.875226 (+ 201us) service_pool.cc:227] Handling call
0805 22:25:15.112442 (+14237216us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0805 22:25:09.651866 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0805 22:25:09.705097 (+ 53231us) write_op.cc:432] Acquiring schema lock in shared mode
0805 22:25:09.705150 (+ 53us) write_op.cc:435] Acquired schema lock
0805 22:25:09.705167 (+ 17us) tablet.cc:660] Decoding operations
0805 22:25:09.876593 (+171426us) write_op.cc:620] Acquiring the partition lock for write op
0805 22:25:09.912532 (+ 35939us) write_op.cc:641] Partition lock acquired for write op
0805 22:25:09.913935 (+ 1403us) tablet.cc:683] Acquiring locks for 10 operations
0805 22:25:10.029622 (+115687us) tablet.cc:699] Row locks acquired
0805 22:25:10.029662 (+ 40us) write_op.cc:260] PREPARE: finished
0805 22:25:10.029858 (+ 196us) write_op.cc:270] Start()
0805 22:25:10.030023 (+ 165us) write_op.cc:276] Timestamp: P: 1785968710029792 usec, L: 0
0805 22:25:10.030063 (+ 40us) op_driver.cc:331] REPLICATION: starting
0805 22:25:10.030807 (+ 744us) log.cc:842] Serialized 4019 byte log entry
0805 22:25:10.309358 (+278551us) op_driver.cc:444] REPLICATION: finished
0805 22:25:10.413466 (+104108us) write_op.cc:301] APPLY: starting
0805 22:25:10.413555 (+ 89us) tablet.cc:1365] starting BulkCheckPresence
0805 22:25:10.413781 (+ 226us) tablet.cc:1368] finished BulkCheckPresence
0805 22:25:10.413891 (+ 110us) tablet.cc:1370] starting ApplyRowOperation cycle
0805 22:25:10.526014 (+112123us) tablet.cc:1381] finished ApplyRowOperation cycle
0805 22:25:10.526367 (+ 353us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=10
0805 22:25:10.526416 (+ 49us) write_op.cc:312] APPLY: finished
0805 22:25:10.526812 (+ 396us) log.cc:842] Serialized 77 byte log entry
0805 22:25:10.526989 (+ 177us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:25:10.527111 (+ 122us) write_op.cc:454] Released schema lock
0805 22:25:10.527193 (+ 82us) write_op.cc:341] FINISH: Updating metrics
0805 22:25:10.527532 (+ 339us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:25:10.527558 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":55629,"reported_tablets":30,"child_traces":[["op",{"apply.queue_time_us":97318,"apply.run_cpu_time_us":4784,"apply.run_wall_time_us":114469,"num_ops":10,"prepare.queue_time_us":344985,"prepare.run_cpu_time_us":31182,"prepare.run_wall_time_us":502583,"raft.queue_time_us":133626,"raft.run_cpu_time_us":16559,"raft.run_wall_time_us":59698,"replication_time_us":279201,"thread_start_us":367810,"threads_started":4,"wal-append.queue_time_us":126971,"wal-append.run_cpu_time_us":36509,"wal-append.run_wall_time_us":1382851}]]}
I20260805 22:25:15.214331 31667 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:15.218245 2452 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260805 22:25:15.218945 2452 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:15.219590 2452 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:15.227738 31667 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:15.228072 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.230485 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.231177 32587 raft_consensus.cc:2454] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:25:15.176483 3650 raft_consensus.cc:3037] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:15.234228 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.237486 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:15.230666 32583 raft_consensus.cc:2454] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:25:15.259218 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.155480 32586 rpcz_store.cc:276] 0805 22:25:12.959311 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:12.959442 (+ 131us) service_pool.cc:227] Handling call
0805 22:25:12.985657 (+ 26215us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:14.913536 (+1927879us) raft_consensus.cc:1520] Early marking committed up to index 6
0805 22:25:15.006313 (+ 92777us) log.cc:842] Serialized 14 byte log entry
0805 22:25:15.141245 (+134932us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:15.141358 (+ 113us) raft_consensus.cc:1631] Marking committed up to 6
0805 22:25:15.141438 (+ 80us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:15.154800 (+ 13362us) spinlock_profiling.cc:243] Waited 1.93 s on lock 0x7b54005613d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:15.154846 (+ 46us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:15.154964 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":144279,"prepare.run_cpu_time_us":33,"prepare.run_wall_time_us":35,"raft.queue_time_us":220,"raft.run_cpu_time_us":135,"raft.run_wall_time_us":13860,"spinlock_wait_cycles":4433087872,"thread_start_us":226933,"threads_started":2}
I20260805 22:25:15.275570 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.279343 3690 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:15.282621 3503 raft_consensus.cc:2781] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:15.283398 3503 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260805 22:25:15.284592 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 56927.7us
I20260805 22:25:15.283883 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:25:15.297667 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 219431us
I20260805 22:25:15.298540 31444 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:15.300578 31444 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:15.302378 32529 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:15.303485 32529 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:15.269213 2855 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:15.306484 32529 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:15.309615 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.310355 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.310391 32529 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:15.313454 3705 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:15.314092 3705 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:15.315104 31713 raft_consensus.cc:2379] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 7: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:15.315444 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.314965 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.316797 31603 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 9.
I20260805 22:25:15.284219 3503 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:15.318086 31667 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:15.318336 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.284168 3690 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:15.322746 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.314630 3705 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:15.323859 2855 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:15.323417 31594 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 10.
I20260805 22:25:15.327028 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.327646 31594 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 9.
I20260805 22:25:15.311934 3605 raft_consensus.cc:2781] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:25:15.330155 3572 consensus_queue.cc:1046] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260805 22:25:15.330942 3605 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:15.330999 3505 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:15.318225 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.332408 31609 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
I20260805 22:25:15.319963 31667 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:15.332595 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.333231 31708 raft_consensus.cc:2454] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:25:15.333586 3361 raft_consensus.cc:2781] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:15.334124 3361 raft_consensus.cc:482] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:15.334748 3361 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:15.335295 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.335887 31594 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
I20260805 22:25:15.320379 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.310170 3588 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260805 22:25:15.338109 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.318598 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.338459 3588 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:15.331557 3572 raft_consensus.cc:3032] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:25:15.333851 31667 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:15.339383 3572 raft_consensus.cc:729] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 6 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:15.339979 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.331640 3605 raft_consensus.cc:3037] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:15.319883 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.331822 3492 raft_consensus.cc:2781] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:15.308022 3697 raft_consensus.cc:2781] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:15.340660 31709 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
I20260805 22:25:15.337815 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.334074 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.368975 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.369762 31708 raft_consensus.cc:2396] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 10 because replica has last-logged OpId of term: 7 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 6 index: 4.
I20260805 22:25:15.318918 32529 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:15.345896 31600 raft_consensus.cc:1261] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260805 22:25:15.349570 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.352550 31667 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:15.307159 3638 raft_consensus.cc:2781] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader election won for term 9
I20260805 22:25:15.405684 3547 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:15.405599 3638 raft_consensus.cc:686] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [term 9 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:15.406469 3547 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:15.407519 3638 consensus_queue.cc:237] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:15.408694 3547 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:15.355017 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.354943 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.355999 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.410421 31601 raft_consensus.cc:2379] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 9: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:15.354943 2855 raft_consensus.cc:2781] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election won for term 9
I20260805 22:25:15.363075 31704 raft_consensus.cc:2379] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:15.366053 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.366911 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.310120 3577 raft_consensus.cc:2781] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:15.370190 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.368846 31714 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 7.
I20260805 22:25:15.371676 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:15.419131 31600 raft_consensus.cc:3037] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 9
I20260805 22:25:15.420703 2855 raft_consensus.cc:686] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:15.425436 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.425675 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:15.426295 32586 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 7
I20260805 22:25:15.426687 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.426888 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.427675 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.427734 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.425452 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:15.428802 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.421681 31603 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:25:15.429453 32589 raft_consensus.cc:3037] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
W20260805 22:25:15.371999 31444 connection.cc:232] RPC call timeout handler was delayed by -1.0293300801422447s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1865s for a response
I20260805 22:25:15.307811 3568 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:15.372723 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 53912.5us
I20260805 22:25:15.373725 3297 raft_consensus.cc:482] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:15.418375 3577 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:15.435204 3297 raft_consensus.cc:504] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:15.435623 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.364563 3505 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:15.376958 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.353621 3588 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:15.368263 3492 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:15.368083 3697 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:15.379979 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.379952 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.381453 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.373664 3696 raft_consensus.cc:2781] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:15.384876 3361 raft_consensus.cc:504] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:15.385565 31554 leader_election.cc:302] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:15.386734 3667 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:15.392028 3651 raft_consensus.cc:504] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:15.370940 3197 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
W20260805 22:25:15.379953 3731 log.cc:924] Time spent T 0191f03fa8984a40afc7ec1e44970c8e P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.102s user 0.000s sys 0.002s
I20260805 22:25:15.395018 3321 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:15.403460 3503 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:15.340639 3572 consensus_queue.cc:260] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:15.404564 3345 raft_consensus.cc:2781] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:15.413661 31667 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:25:15.472309 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 58772.2us
I20260805 22:25:15.456485 3493 raft_consensus.cc:2781] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260805 22:25:15.472977 31608 raft_consensus.cc:2379] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:15.473439 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.474601 31601 raft_consensus.cc:1226] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 9. Current term is 10. Ops: []
I20260805 22:25:15.410458 31602 raft_consensus.cc:2379] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 11: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:15.429406 31714 raft_consensus.cc:2454] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:25:15.430696 32529 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:15.376288 2452 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:15.429275 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:25:15.430217 31444 connection.cc:232] RPC call timeout handler was delayed by -1.0230300801340491s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.18720000000000003s for a response
I20260805 22:25:15.446501 3696 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:15.478367 31444 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:15.436777 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.442466 31609 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 6.
W20260805 22:25:15.446547 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 61054.4us
I20260805 22:25:15.451983 32589 raft_consensus.cc:2454] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:25:15.452804 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.484632 31708 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 9.
I20260805 22:25:15.439493 3577 raft_consensus.cc:3037] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:15.456511 31596 raft_consensus.cc:2379] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 12: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:15.488386 31704 raft_consensus.cc:2379] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 8: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:15.460821 31668 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:15.442148 3492 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:15.469251 31600 raft_consensus.cc:2454] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 9.
I20260805 22:25:15.443645 3697 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Advancing to term 8
W20260805 22:25:15.469978 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 395914us
I20260805 22:25:15.486596 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.477136 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.498377 31603 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 10.
I20260805 22:25:15.477751 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.502648 31444 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:15.491308 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.503947 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.490068 31668 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:15.513592 31556 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:15.514269 31597 raft_consensus.cc:2379] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 7: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:15.504755 31714 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 10.
I20260805 22:25:15.518903 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.519649 31603 raft_consensus.cc:2454] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 10.
I20260805 22:25:15.500627 31554 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:15.521260 31594 raft_consensus.cc:1226] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 12 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 11. Current term is 12. Ops: []
I20260805 22:25:15.522154 31444 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:15.471894 3345 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:15.522346 31556 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:15.567349 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 66786.1us
I20260805 22:25:15.444375 3708 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:15.512353 3707 raft_consensus.cc:2781] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:15.571326 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.571336 3707 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:15.572139 3707 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:15.527580 31444 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:15.572233 31596 raft_consensus.cc:2379] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 10: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:15.575997 3361 leader_election.cc:288] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607)
I20260805 22:25:15.520311 2956 raft_consensus.cc:2781] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:15.576717 2956 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:15.577088 2956 raft_consensus.cc:3037] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:15.583542 31492 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1962) took 1021 ms. Trace:
I20260805 22:25:15.573527 3684 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260805 22:25:15.574203 3736 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260805 22:25:15.523355 3499 raft_consensus.cc:2781] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:15.583922 31492 rpcz_store.cc:276] 0805 22:25:14.561717 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:15.038545 (+476828us) service_pool.cc:227] Handling call
0805 22:25:15.038852 (+ 307us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:15.063939 (+ 25087us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:15.063965 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:15.064376 (+ 411us) log.cc:842] Serialized 26 byte log entry
0805 22:25:15.304751 (+240375us) raft_consensus.cc:1631] Marking committed up to 5
0805 22:25:15.304808 (+ 57us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:15.305805 (+ 997us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:15.583321 (+277516us) raft_consensus.cc:1671] finished
0805 22:25:15.583338 (+ 17us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:15.583485 (+ 147us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24707,"raft.queue_time_us":25392,"raft.run_cpu_time_us":159,"raft.run_wall_time_us":21398,"thread_start_us":220416,"threads_started":1,"wal-append.queue_time_us":502140}
W20260805 22:25:15.583448 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 61181.7us
I20260805 22:25:15.525733 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:15.586208 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:15.493525 3328 raft_consensus.cc:2781] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:15.462384 3197 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:15.599053 29789 debug-util.cc:398] Leaking SignalData structure 0x7b080000eec0 after lost signal to thread 31668
I20260805 22:25:15.596381 31556 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:15.474318 3493 raft_consensus.cc:482] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:15.597119 31603 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:15.617357 31596 raft_consensus.cc:1261] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 6 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
W20260805 22:25:15.601063 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 85435us
I20260805 22:25:15.615427 31556 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:15.613139 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
W20260805 22:25:15.616586 31486 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.563s user 0.010s sys 0.022s
W20260805 22:25:15.625736 31486 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.573s user 0.011s sys 0.023s
I20260805 22:25:15.625941 31601 raft_consensus.cc:2379] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:15.626502 31486 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1963) took 1064 ms. Trace:
I20260805 22:25:15.588369 3345 raft_consensus.cc:3037] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:15.489949 3696 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:15.626885 31486 rpcz_store.cc:276] 0805 22:25:14.562007 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:15.051866 (+489859us) service_pool.cc:227] Handling call
0805 22:25:15.052158 (+ 292us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:15.052332 (+ 174us) raft_consensus.cc:1520] Early marking committed up to index 4
0805 22:25:15.052646 (+ 314us) log.cc:842] Serialized 14 byte log entry
0805 22:25:15.626181 (+573535us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:15.626232 (+ 51us) raft_consensus.cc:1631] Marking committed up to 4
0805 22:25:15.626276 (+ 44us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:15.626337 (+ 61us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:15.626451 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":196,"prepare.run_cpu_time_us":37,"prepare.run_wall_time_us":38,"thread_start_us":573185,"threads_started":1}
I20260805 22:25:15.489037 3505 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Leader pre-election won for term 12
I20260805 22:25:15.628041 31706 raft_consensus.cc:2379] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 7: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:15.522385 3572 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:15.514539 3297 leader_election.cc:288] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:15.595261 3684 raft_consensus.cc:482] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:15.599538 3499 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:15.612939 3328 raft_consensus.cc:482] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:15.595754 3496 raft_consensus.cc:2726] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:15.469937 2855 consensus_queue.cc:237] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 7.4, Last appended by leader: 4, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:15.595547 3736 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:15.597952 3235 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260805 22:25:15.494313 3547 consensus_queue.cc:1035] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.144s
I20260805 22:25:15.629807 3493 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Advancing to term 11
W20260805 22:25:15.657120 31447 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.562s user 0.009s sys 0.008s
W20260805 22:25:15.679986 31447 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.585s user 0.009s sys 0.009s
I20260805 22:25:15.581172 3707 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:15.677800 31603 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:15.678259 32586 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:15.627748 3690 raft_consensus.cc:2781] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:15.635080 3505 raft_consensus.cc:482] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:15.685195 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.655277 3684 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:15.686795 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.655844 3499 raft_consensus.cc:3037] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:15.689118 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:15.741053 31486 raft_consensus.cc:3037] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:15.689729 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.690250 3707 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:15.691135 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.692128 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.693089 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.684530 3496 raft_consensus.cc:2741] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election decision vote started in defunct term 6: won
I20260805 22:25:15.688553 3705 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:15.695571 2956 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:15.705819 3297 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:15.706475 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.758208 31492 raft_consensus.cc:2379] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 9: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:15.686515 3505 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Advancing to term 12
I20260805 22:25:15.724953 32581 raft_consensus.cc:1261] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 5 index: 6. Preceding OpId from leader: term: 6 index: 6. (term mismatch)
I20260805 22:25:15.758970 32581 pending_rounds.cc:76] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf: Aborting all ops after (but not including) 5
I20260805 22:25:15.658625 3328 raft_consensus.cc:3037] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:15.760686 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.761662 31495 raft_consensus.cc:2396] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 12 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 6.
I20260805 22:25:15.676054 3736 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:15.737711 31490 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1968) took 1174 ms. Trace:
I20260805 22:25:15.738354 3651 leader_election.cc:288] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:15.776042 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.685945 31493 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:15.654620 3527 raft_consensus.cc:2726] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260805 22:25:15.742527 31494 raft_consensus.cc:2454] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:25:15.742483 31502 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:15.741873 31499 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 4.
I20260805 22:25:15.778987 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.679467 3235 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:15.735450 3708 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:15.759214 32581 pending_rounds.cc:97] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf: Aborting uncommitted NO_OP operation due to leader change: 5.6
I20260805 22:25:15.786273 32581 raft_consensus.cc:2866] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:25:15.773885 31490 rpcz_store.cc:276] 0805 22:25:14.563299 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:15.054715 (+491416us) service_pool.cc:227] Handling call
0805 22:25:15.071583 (+ 16868us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:15.084945 (+ 13362us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:15.084972 (+ 27us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:15.137759 (+ 52787us) log.cc:842] Serialized 26 byte log entry
0805 22:25:15.514880 (+377121us) raft_consensus.cc:1631] Marking committed up to 3
0805 22:25:15.514915 (+ 35us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:15.514979 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:15.737478 (+222499us) raft_consensus.cc:1671] finished
0805 22:25:15.737500 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:15.737642 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":34182,"raft.queue_time_us":21373,"raft.run_cpu_time_us":140,"raft.run_wall_time_us":25526,"thread_start_us":376602,"threads_started":1,"wal-append.queue_time_us":520144}
I20260805 22:25:15.778889 32586 raft_consensus.cc:1226] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 9 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 8. Current term is 9. Ops: []
I20260805 22:25:15.740895 3568 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 12 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:15.780445 31495 raft_consensus.cc:2379] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:15.780339 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.792596 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.793255 31502 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 8.
I20260805 22:25:15.794142 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.795197 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.796255 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.788697 3235 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:15.800922 3605 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:15.802235 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:15.802850 31499 raft_consensus.cc:2362] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6. Re-sending same reply.
I20260805 22:25:15.803257 3605 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:15.804123 3605 raft_consensus.cc:2781] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:15.804477 3605 raft_consensus.cc:482] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:15.804850 3605 raft_consensus.cc:3037] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:15.803671 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.808288 31499 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:15.809263 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.809971 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.810418 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.810621 31493 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 7.
I20260805 22:25:15.811040 31499 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:25:15.811947 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.813311 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.813885 31499 raft_consensus.cc:2454] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:25:15.814270 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.814725 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.815778 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.797243 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.817631 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.824995 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.826550 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.833055 31486 raft_consensus.cc:2454] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 9.
I20260805 22:25:15.834756 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.779228 3690 raft_consensus.cc:482] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:15.843873 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.845607 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.843596 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.847263 31490 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 8.
I20260805 22:25:15.850484 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.852876 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.790628 3321 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:15.817296 2956 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:15.717267 3697 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:15.862272 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 334753us
I20260805 22:25:15.865150 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.789537 3503 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:15.866015 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 281486us
W20260805 22:25:15.881661 3584 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.609s user 0.064s sys 0.074s
I20260805 22:25:15.872674 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.900830 31492 raft_consensus.cc:2454] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:25:15.901132 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.873420 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.825730 2452 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 11 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:15.876911 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.851475 3690 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:15.836647 3667 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:15.919268 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.880472 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.920140 31493 raft_consensus.cc:2379] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 9: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:15.882051 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.893950 31444 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
W20260805 22:25:15.893966 31357 thread.cc:636] prepare [worker] (thread pool) Time spent creating pthread: real 0.522s user 0.022s sys 0.019s
I20260805 22:25:15.896459 31554 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:15.762176 3297 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:15.900534 3584 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.630s user 0.064s sys 0.074s
W20260805 22:25:15.913900 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 483255us
I20260805 22:25:15.927258 32529 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:15.928133 32529 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
W20260805 22:25:15.903668 3621 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.527s user 0.000s sys 0.042s
I20260805 22:25:15.901748 3197 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:15.908741 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.921120 31499 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 10.
W20260805 22:25:15.929277 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 313925us
I20260805 22:25:15.918322 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.932225 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:15.935401 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:25:15.923336 31357 thread.cc:602] prepare [worker] (thread pool) Time spent starting thread: real 0.570s user 0.023s sys 0.019s
I20260805 22:25:15.938735 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.936040 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:15.943303 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.948145 31714 raft_consensus.cc:2379] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:15.949281 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.948325 31704 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:15.944622 31556 leader_election.cc:302] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994
I20260805 22:25:15.951586 31492 raft_consensus.cc:2454] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
W20260805 22:25:15.935021 3621 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.560s user 0.000s sys 0.042s
I20260805 22:25:15.952250 31494 raft_consensus.cc:2379] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:15.952608 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:15.965620 31706 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Advancing to term 12
I20260805 22:25:15.896061 3492 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:15.929970 3496 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260805 22:25:15.969324 3297 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:15.994292 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:25:15.969910 3745 log.cc:924] Time spent T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.068s user 0.000s sys 0.000s
I20260805 22:25:15.970507 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:16.034904 31499 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 4.
W20260805 22:25:15.973987 3748 log.cc:924] Time spent T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260805 22:25:15.940485 3527 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:15.975867 3505 raft_consensus.cc:504] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 12 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:16.043181 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.929811 3746 raft_consensus.cc:2781] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:16.031219 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:16.000082 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:16.033500 31714 raft_consensus.cc:1226] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:16.050371 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:16.049722 31704 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 10.
I20260805 22:25:15.994103 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:15.953032 3705 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 9 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:15.982514 3588 consensus_meta.cc:287] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.542s user 0.007s sys 0.003s
I20260805 22:25:15.949249 3235 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:15.966617 3650 consensus_meta.cc:287] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.688s user 0.007s sys 0.005s
I20260805 22:25:16.039386 31444 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:16.040786 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:16.043915 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:16.073190 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:16.074472 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:16.076757 31715 raft_consensus.cc:2379] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 10: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:16.070773 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:16.080663 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:16.081869 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:16.065327 3496 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:16.033520 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.997677 31501 raft_consensus.cc:1261] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 5 index: 5. (index mismatch)
I20260805 22:25:16.068488 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:16.077777 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:25:16.084635 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 45340us
I20260805 22:25:16.094162 31705 raft_consensus.cc:2379] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 7: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:16.100020 31715 raft_consensus.cc:2454] T 9f41c2ff98f049c3bc654aa98cd77694 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
I20260805 22:25:16.094841 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:16.095185 31711 raft_consensus.cc:2379] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 8: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:16.100472 31444 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:16.049327 3746 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:16.064260 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 11 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:16.101943 31714 raft_consensus.cc:2354] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 11. Current term is 12.
I20260805 22:25:16.073964 31492 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:25:15.981299 3499 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:16.098560 31707 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 9.
I20260805 22:25:16.103114 31444 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:16.112864 31500 raft_consensus.cc:1226] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 9. Current term is 10. Ops: []
I20260805 22:25:16.113596 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:15.982951 3759 consensus_queue.cc:1035] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.111s
I20260805 22:25:16.117715 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:16.069854 3572 raft_consensus.cc:2781] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:25:16.117201 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:16.122556 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:16.123417 31708 raft_consensus.cc:2379] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 9: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:16.093894 3760 raft_consensus.cc:2781] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260805 22:25:16.143388 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 194309us
I20260805 22:25:16.103381 3496 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:16.048978 3527 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:16.092859 3197 raft_consensus.cc:2781] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
W20260805 22:25:16.128904 3577 consensus_meta.cc:287] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.608s user 0.010s sys 0.000s
I20260805 22:25:16.138741 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:25:16.140075 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 211998us
I20260805 22:25:16.094424 3756 raft_consensus.cc:2781] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:25:16.118155 3232 raft_consensus.cc:2781] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:16.052577 3328 raft_consensus.cc:504] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:16.102386 3746 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:16.079524 3650 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:16.075294 3588 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:16.254065 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 309500us
I20260805 22:25:16.180791 31705 raft_consensus.cc:1226] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 9. Current term is 10. Ops: []
I20260805 22:25:16.243088 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:25:16.261333 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 120699us
I20260805 22:25:16.261405 31556 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:16.262060 31554 leader_election.cc:302] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:16.275497 31711 raft_consensus.cc:1261] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 10 index: 7. (index mismatch)
I20260805 22:25:16.261853 32529 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:16.276386 31501 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 8.
W20260805 22:25:16.280819 31668 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.639s user 0.011s sys 0.053s
W20260805 22:25:16.291231 31668 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.649s user 0.013s sys 0.054s
I20260805 22:25:16.280210 31502 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 10.
I20260805 22:25:16.127475 3345 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:16.291846 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:16.281415 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
W20260805 22:25:16.291754 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 672318us
I20260805 22:25:16.279982 31602 raft_consensus.cc:1261] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 9 index: 9. (index mismatch)
I20260805 22:25:16.293473 3780 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:16.294021 31668 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:16.294214 3780 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:16.294766 3780 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:16.293495 31603 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:16.295634 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:16.296285 31601 raft_consensus.cc:2454] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 5.
I20260805 22:25:16.297195 3345 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:16.297909 31602 raft_consensus.cc:1261] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 7 index: 5. (index mismatch)
I20260805 22:25:16.297676 31713 raft_consensus.cc:2379] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:16.275106 31556 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:16.278362 3577 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:16.143790 3297 raft_consensus.cc:2781] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader election won for term 9
I20260805 22:25:16.319239 31603 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 10.
I20260805 22:25:16.279938 3764 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260805 22:25:16.249480 3758 log.cc:924] Time spent T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.153s user 0.001s sys 0.000s
W20260805 22:25:16.301324 3754 log.cc:924] Time spent T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.158s user 0.000s sys 0.000s
I20260805 22:25:16.277923 3736 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:16.301913 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:16.333047 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:16.302721 3759 consensus_queue.cc:1046] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 11.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260805 22:25:16.303311 3777 raft_consensus.cc:2781] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader election won for term 10
I20260805 22:25:16.334633 3777 raft_consensus.cc:686] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 10 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:16.335632 3777 consensus_queue.cc:237] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:16.336956 3736 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:16.161317 3572 raft_consensus.cc:686] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:16.328789 3764 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:16.337765 3764 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:16.338292 31706 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 12.
I20260805 22:25:16.338078 3572 consensus_queue.cc:237] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:16.338943 31715 raft_consensus.cc:1226] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 12 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 11. Current term is 12. Ops: []
I20260805 22:25:16.180683 3197 raft_consensus.cc:482] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:16.341506 3197 raft_consensus.cc:3037] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
W20260805 22:25:16.341930 3772 log.cc:924] Time spent T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.077s user 0.000s sys 0.000s
I20260805 22:25:16.343253 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:16.343858 31596 raft_consensus.cc:3037] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:16.141611 3760 raft_consensus.cc:482] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:16.345765 3760 raft_consensus.cc:3037] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:16.348627 3588 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:16.265565 3696 consensus_meta.cc:287] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.611s user 0.009s sys 0.005s
W20260805 22:25:16.343267 3752 log.cc:924] Time spent T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.241s user 0.001s sys 0.000s
W20260805 22:25:16.349642 3770 log.cc:924] Time spent T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.075s user 0.001s sys 0.000s
I20260805 22:25:16.317260 31556 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:16.318459 3773 log.cc:924] Time spent T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.055s user 0.000s sys 0.000s
I20260805 22:25:16.333161 3505 leader_election.cc:288] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 12 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:16.182371 3697 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:16.162484 3321 consensus_queue.cc:1046] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.144s
I20260805 22:25:16.263847 3756 raft_consensus.cc:482] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:16.351294 3756 raft_consensus.cc:3037] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:16.349669 3696 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:16.320652 2855 consensus_queue.cc:1035] T 36ee93639c3f4084b94145c6ee60a2a8 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20260805 22:25:16.350018 31495 raft_consensus.cc:1226] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 9. Current term is 10. Ops: []
I20260805 22:25:16.323577 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:16.323845 3527 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:16.324404 3235 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:16.326032 3328 leader_election.cc:288] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:16.300175 32529 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:16.328080 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:16.316334 2452 raft_consensus.cc:2726] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:25:16.334003 31603 raft_consensus.cc:2379] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 9: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:16.330943 3577 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:16.355031 31502 raft_consensus.cc:2454] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:25:16.356488 31556 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:16.367478 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 68258.9us
I20260805 22:25:16.356814 3664 raft_consensus.cc:2726] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:16.364714 31711 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 9.
I20260805 22:25:16.370343 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:16.371497 31556 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:16.372493 31556 leader_election.cc:302] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:16.363365 31596 raft_consensus.cc:2454] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:25:16.366048 31554 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:16.363327 31668 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:16.374055 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:16.375275 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:16.375808 31609 raft_consensus.cc:3037] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:16.376142 31667 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:16.376396 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:16.321291 3776 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260805 22:25:16.378161 31596 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Advancing to term 12
I20260805 22:25:16.379477 31668 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:25:16.368882 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 68778.7us
I20260805 22:25:16.379565 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:16.380805 31602 raft_consensus.cc:3037] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:16.384598 32529 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:16.354270 3297 raft_consensus.cc:686] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [term 9 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:16.364537 3588 raft_consensus.cc:3032] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 11 LEADER]: Stepping down as leader of term 11
I20260805 22:25:16.390385 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:16.394867 31667 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:16.396327 31667 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:16.397651 2855 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260805 22:25:16.398008 2855 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:16.398339 2855 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:16.390810 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:16.364380 3696 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:16.393499 31608 raft_consensus.cc:1261] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260805 22:25:16.399971 32589 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:16.402832 32587 raft_consensus.cc:1261] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 6. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260805 22:25:16.398098 31444 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:16.378804 3667 raft_consensus.cc:2781] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:16.405668 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:16.405742 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:16.404137 31444 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:16.381788 3776 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:16.407596 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260805 22:25:16.393033 3493 consensus_meta.cc:287] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.650s user 0.007s sys 0.004s
I20260805 22:25:16.362772 3690 raft_consensus.cc:504] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:16.381664 3577 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260805 22:25:16.425767 31444 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:16.394279 3345 raft_consensus.cc:2781] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:25:16.384166 2452 raft_consensus.cc:2781] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:25:16.390039 3664 raft_consensus.cc:2781] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:16.427248 31602 raft_consensus.cc:2454] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:16.427485 31444 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:16.433248 31444 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:16.405320 3769 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:16.394011 3499 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:16.408066 3667 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:16.438169 31492 raft_consensus.cc:2379] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 8: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:16.438683 31604 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59028 (request call id 2481) took 1088 ms. Trace:
I20260805 22:25:16.439126 31604 rpcz_store.cc:276] 0805 22:25:15.350292 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:15.411132 (+ 60840us) service_pool.cc:227] Handling call
0805 22:25:15.411484 (+ 352us) raft_consensus.cc:1457] Updating replica for 2 ops
0805 22:25:15.411827 (+ 343us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:15.411872 (+ 45us) raft_consensus.cc:1525] Triggering prepare for 2 ops
0805 22:25:15.412474 (+ 602us) log.cc:842] Serialized 52 byte log entry
0805 22:25:15.893033 (+480559us) raft_consensus.cc:1631] Marking committed up to 10
0805 22:25:16.119984 (+226951us) log.cc:842] Serialized 14 byte log entry
0805 22:25:16.121424 (+ 1440us) log.cc:842] Serialized 14 byte log entry
0805 22:25:16.129704 (+ 8280us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:16.129786 (+ 82us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:16.438443 (+308657us) raft_consensus.cc:1671] finished
0805 22:25:16.438465 (+ 22us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:16.438625 (+ 160us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7989,"prepare.queue_time_us":64525,"prepare.run_cpu_time_us":76,"prepare.run_wall_time_us":78,"raft.queue_time_us":24967,"raft.run_cpu_time_us":425,"raft.run_wall_time_us":58550,"thread_start_us":683958,"threads_started":2,"wal-append.queue_time_us":649889}
I20260805 22:25:16.413614 3776 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:16.427817 32589 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:25:16.421964 31596 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 12.
I20260805 22:25:16.394589 3588 raft_consensus.cc:729] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 11 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:16.434223 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:16.428299 3577 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:16.428519 3584 raft_consensus.cc:2726] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:16.428023 31554 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:16.409984 3759 consensus_queue.cc:1046] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.034s
I20260805 22:25:16.410451 3321 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:16.422161 3696 raft_consensus.cc:2726] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:16.396488 3527 raft_consensus.cc:2781] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260805 22:25:16.409884 3651 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 12 FOLLOWER]: Leader election won for term 12
I20260805 22:25:16.429427 3328 raft_consensus.cc:2726] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:16.441121 3592 raft_consensus.cc:2726] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:16.438221 3650 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:16.434027 3664 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:16.395979 3496 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:16.437767 3281 raft_consensus.cc:2781] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:16.435694 31444 leader_election.cc:398] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 11 pre-election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 11. Current term is 12.
W20260805 22:25:16.448443 3684 consensus_meta.cc:287] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c: Time spent flushing consensus metadata: real 0.689s user 0.009s sys 0.003s
I20260805 22:25:16.453647 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:16.426967 3493 raft_consensus.cc:504] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:16.432353 3765 raft_consensus.cc:2781] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260805 22:25:16.434165 3345 raft_consensus.cc:686] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:16.438078 2452 raft_consensus.cc:686] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:16.441675 3667 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:16.473784 31444 leader_election.cc:401] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 11 pre-election: Cancelling election due to peer responding with higher term
I20260805 22:25:16.475258 31501 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 8.
W20260805 22:25:16.450115 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 65628.6us
I20260805 22:25:16.537961 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:25:16.503473 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 107191us
I20260805 22:25:16.674822 31711 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
mode: GRACEFUL
new_leader_uuid: "f8e9f4c756da4203917c464e2b179994"
from {username='slave'} at 127.0.0.1:43958
I20260805 22:25:16.573333 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
W20260805 22:25:16.719519 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 283933us
I20260805 22:25:16.719560 32529 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:16.607812 31609 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:25:16.722600 31494 raft_consensus.cc:3037] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:16.532892 31601 raft_consensus.cc:1226] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 10. Current term is 11. Ops: [8.6-10.7]
I20260805 22:25:16.577693 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:16.723359 32529 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:16.581705 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:16.676064 31668 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:16.642747 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:16.476027 2855 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:16.709182 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 336701us
I20260805 22:25:16.725509 31711 raft_consensus.cc:595] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Received request to transfer leadership to TS f8e9f4c756da4203917c464e2b179994
I20260805 22:25:16.459105 3664 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:16.455387 3638 raft_consensus.cc:3032] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 9 LEADER]: Stepping down as leader of term 9
I20260805 22:25:16.445019 3577 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:16.575199 3496 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 11 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:16.473309 3651 raft_consensus.cc:686] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 12 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:16.726152 3791 raft_consensus.cc:2726] T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:16.726020 3705 raft_consensus.cc:2781] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:16.746239 31667 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:25:16.575378 3605 consensus_meta.cc:287] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.770s user 0.010s sys 0.003s
I20260805 22:25:16.574195 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:16.593812 3584 raft_consensus.cc:3032] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:25:16.728619 32529 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:16.761111 32529 leader_election.cc:302] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:16.763396 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:16.730718 2855 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 11 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:16.764364 2855 raft_consensus.cc:2781] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election won for term 8
I20260805 22:25:16.764778 2855 raft_consensus.cc:686] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:16.451756 3527 raft_consensus.cc:482] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
W20260805 22:25:16.730751 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 54751.8us
W20260805 22:25:16.578714 3782 log.cc:924] Time spent T 00000000000000000000000000000000 P 28a5659dcb9347329aa3accbe95f66b0: Append to log took a long time: real 0.140s user 0.001s sys 0.000s
I20260805 22:25:16.742326 31603 raft_consensus.cc:2379] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 11: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
W20260805 22:25:16.566525 3778 log.cc:924] Time spent T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.168s user 0.000s sys 0.000s
I20260805 22:25:16.451336 3232 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:16.725996 3759 raft_consensus.cc:3037] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Advancing to term 12
I20260805 22:25:16.474808 3281 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:16.473690 3786 raft_consensus.cc:2741] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader pre-election decision vote started in defunct term 9: won
I20260805 22:25:16.760596 3584 raft_consensus.cc:729] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:16.726703 2452 consensus_queue.cc:237] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:16.572919 31447 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:16.570673 3756 raft_consensus.cc:504] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:16.759913 3705 raft_consensus.cc:482] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:16.758953 31492 raft_consensus.cc:2379] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 10: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:16.578862 3492 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:16.593191 3690 leader_election.cc:288] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:16.572554 3764 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:16.776985 2855 consensus_queue.cc:237] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.4, Last appended by leader: 4, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:16.782691 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:16.783453 31609 raft_consensus.cc:2354] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 10. Current term is 11.
I20260805 22:25:16.761368 31601 raft_consensus.cc:1226] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:16.765565 3697 raft_consensus.cc:2781] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:16.765926 3771 raft_consensus.cc:2741] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader pre-election decision vote started in defunct term 7: won
I20260805 22:25:16.458204 3780 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:16.720683 3765 raft_consensus.cc:482] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:16.734865 3638 raft_consensus.cc:729] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:16.777532 3281 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:16.771762 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:16.676311 2956 consensus_queue.cc:1035] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.086s
I20260805 22:25:16.502791 3684 raft_consensus.cc:504] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:16.762100 32529 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:16.448899 3297 consensus_queue.cc:237] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 7.5, Last appended by leader: 5, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:16.584681 3776 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:16.576411 3760 raft_consensus.cc:504] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:16.593981 3769 raft_consensus.cc:2726] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:25:16.741657 3321 consensus_queue.cc:1035] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 4, Time since last communication: 0.034s
I20260805 22:25:16.741422 3328 consensus_queue.cc:1046] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.039s
I20260805 22:25:16.769915 3527 raft_consensus.cc:3037] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
W20260805 22:25:16.776738 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 119395us
I20260805 22:25:16.577330 3197 raft_consensus.cc:504] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:16.729360 31711 raft_consensus.cc:603] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:25:16.539114 3588 consensus_queue.cc:260] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 11, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:16.789862 3493 leader_election.cc:288] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 11 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:16.790381 3651 consensus_queue.cc:237] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 11.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:16.775750 3232 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:16.757159 31444 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:16.761775 3605 raft_consensus.cc:504] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:16.779608 3705 raft_consensus.cc:3037] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:16.583204 3746 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:16.741770 3621 raft_consensus.cc:2781] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election won for term 8
I20260805 22:25:16.791847 3736 raft_consensus.cc:2781] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:25:16.804517 31601 raft_consensus.cc:1226] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 7. Current term is 8. Ops: []
I20260805 22:25:16.794269 3697 raft_consensus.cc:482] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:16.798153 31603 consensus_queue.cc:237] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 2, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } }
I20260805 22:25:16.792217 3235 consensus_queue.cc:1035] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.041s
I20260805 22:25:16.913281 31600 raft_consensus.cc:1226] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [term 10 LEADER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 9. Current term is 10. Ops: []
I20260805 22:25:16.914963 3235 raft_consensus.cc:2781] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260805 22:25:16.915381 3235 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:16.915733 3235 raft_consensus.cc:3037] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:16.777544 3345 consensus_queue.cc:237] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 5.7, Last appended by leader: 7, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:16.908146 3736 raft_consensus.cc:482] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:16.917807 3736 raft_consensus.cc:3037] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Advancing to term 5
W20260805 22:25:16.820987 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 58951.9us
I20260805 22:25:16.849327 31597 raft_consensus.cc:2354] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 9. Current term is 10.
I20260805 22:25:16.808688 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:16.928979 31609 raft_consensus.cc:2379] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 8: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:16.911023 3697 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:16.912422 3605 leader_election.cc:288] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:16.943594 31494 raft_consensus.cc:2454] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
W20260805 22:25:16.907485 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 150396us
W20260805 22:25:16.938835 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 192692us
I20260805 22:25:16.945194 31667 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:16.855486 31594 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 10.
I20260805 22:25:16.945312 31444 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:16.939169 31711 raft_consensus.cc:1261] T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
W20260805 22:25:16.942736 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 369884us
I20260805 22:25:16.945240 32529 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:16.949507 31447 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:16.848294 3696 raft_consensus.cc:2726] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:25:16.890466 3756 leader_election.cc:288] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:16.954178 31667 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:16.892804 3764 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:16.964452 31667 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 12 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:16.891736 3780 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:16.957952 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:16.966641 31706 raft_consensus.cc:2379] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 11: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:16.848047 3771 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:16.961979 32529 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:16.796618 3765 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:16.934903 3621 raft_consensus.cc:686] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:16.971547 2855 raft_consensus.cc:2781] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 12 FOLLOWER]: Leader election won for term 12
I20260805 22:25:16.972505 31447 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:16.951656 3592 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader election won for term 10
I20260805 22:25:16.975633 2855 raft_consensus.cc:686] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 12 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:16.957284 3505 raft_consensus.cc:2726] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260805 22:25:16.977169 31447 leader_election.cc:302] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:16.964274 3605 raft_consensus.cc:2726] T 4172338662524cb29310f48e0861873e P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:16.980340 2855 consensus_queue.cc:237] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 10.7, Last appended by leader: 7, Current term: 12, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:16.943531 3503 consensus_queue.cc:1035] T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 5, Time since last communication: 0.028s
I20260805 22:25:16.965175 3790 raft_consensus.cc:482] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:16.985113 31706 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:43958
I20260805 22:25:16.985574 3499 consensus_queue.cc:1035] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:25:16.983081 3780 raft_consensus.cc:2781] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:25:16.894357 3584 consensus_queue.cc:260] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:16.967612 3490 consensus_queue.cc:1035] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.023s
I20260805 22:25:16.986749 3760 leader_election.cc:288] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:16.991528 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:16.982770 32529 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:16.996408 31706 raft_consensus.cc:595] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 LEADER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:16.989790 3684 leader_election.cc:288] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:16.964278 3800 consensus_queue.cc:1035] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.050s
I20260805 22:25:16.974825 3771 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:17.101887 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 79377.5us
I20260805 22:25:17.116868 32594 raft_consensus.cc:1226] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 7. Current term is 8. Ops: []
W20260805 22:25:17.097023 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 148276us
I20260805 22:25:17.285776 31707 raft_consensus.cc:1261] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260805 22:25:17.078215 3818 raft_consensus.cc:2726] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:25:17.286366 3605 raft_consensus.cc:2726] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:16.975175 3803 raft_consensus.cc:2781] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election won for term 7
W20260805 22:25:17.288228 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 305499us
I20260805 22:25:17.281683 31711 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Advancing to term 10
W20260805 22:25:17.290390 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 325994us
W20260805 22:25:17.171067 3817 log.cc:924] Time spent T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
W20260805 22:25:17.105409 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 128286us
I20260805 22:25:17.296757 32529 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:17.156622 3795 log.cc:924] Time spent T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.155s user 0.000s sys 0.001s
I20260805 22:25:17.286728 3802 consensus_queue.cc:1046] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 7.7, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20260805 22:25:17.052768 3328 raft_consensus.cc:2726] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:17.301326 3696 raft_consensus.cc:2726] T dc38ff59fd654f6689e3a88fdbb765cf P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:25:17.303900 3756 consensus_queue.cc:1035] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260805 22:25:16.998834 3804 raft_consensus.cc:2726] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:17.293478 31444 leader_election.cc:398] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 election: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 10. Current term is 11.
I20260805 22:25:17.298477 31667 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:17.114444 3588 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Advancing to term 12
I20260805 22:25:17.237535 3746 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:17.282543 3780 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:17.326004 3780 raft_consensus.cc:3037] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:17.115427 3705 raft_consensus.cc:504] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:17.116343 3736 raft_consensus.cc:504] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:17.328979 3736 leader_election.cc:288] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:17.172652 3235 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:17.173421 3197 leader_election.cc:288] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:17.330928 3235 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:17.312970 31707 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
mode: GRACEFUL
new_leader_uuid: "f8e9f4c756da4203917c464e2b179994"
from {username='slave'} at 127.0.0.1:43958
I20260805 22:25:17.331833 31707 raft_consensus.cc:595] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 12 FOLLOWER]: Received request to transfer leadership to TS f8e9f4c756da4203917c464e2b179994
I20260805 22:25:17.332170 31707 raft_consensus.cc:603] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 12 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:25:17.315236 31444 leader_election.cc:401] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 election: Cancelling election due to peer responding with higher term
I20260805 22:25:17.318077 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:16.999670 3621 consensus_queue.cc:237] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:17.334780 3705 leader_election.cc:288] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:17.138676 3664 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:17.317287 3281 consensus_meta.cc:287] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.508s user 0.007s sys 0.003s
I20260805 22:25:17.321777 2956 raft_consensus.cc:3032] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 7 LEADER]: Stepping down as leader of term 7
I20260805 22:25:17.297056 3527 raft_consensus.cc:504] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:17.295969 31447 leader_election.cc:302] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:17.291859 3592 raft_consensus.cc:686] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 10 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:17.059388 3321 consensus_queue.cc:1046] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.298s
I20260805 22:25:17.340689 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:17.341004 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:17.341284 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:17.346371 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
W20260805 22:25:17.350657 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 57325.4us
I20260805 22:25:17.361524 31601 raft_consensus.cc:2379] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 6: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:17.362900 31444 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:17.352072 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:17.352547 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:17.364138 31604 raft_consensus.cc:3037] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 3 FOLLOWER]: Advancing to term 4
I20260805 22:25:17.353034 31596 raft_consensus.cc:1226] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:17.353252 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:17.365440 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:17.132793 3577 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:17.106770 3667 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:17.297870 3803 raft_consensus.cc:686] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
W20260805 22:25:17.355800 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 59895us
I20260805 22:25:17.358194 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:17.367882 31609 raft_consensus.cc:2379] T fe370453251f4300a8d90c397021595c P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 8: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:17.367321 31602 raft_consensus.cc:2379] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 8: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:17.005029 3790 raft_consensus.cc:504] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:17.339802 3493 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:17.113207 3638 consensus_queue.cc:260] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:17.357403 3235 raft_consensus.cc:2726] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:25:17.350193 2956 raft_consensus.cc:729] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:17.368952 31447 leader_election.cc:302] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:17.422794 31604 raft_consensus.cc:2454] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:25:17.356845 3197 raft_consensus.cc:3037] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:17.319217 3697 raft_consensus.cc:504] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:17.350018 3281 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:17.428772 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 57366.6us
W20260805 22:25:17.512760 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 196763us
I20260805 22:25:17.500689 31500 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:40728 (request call id 1994) took 1137 ms. Trace:
W20260805 22:25:17.441808 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 106938us
W20260805 22:25:17.511677 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 142792us
I20260805 22:25:17.512269 31444 leader_election.cc:398] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 9. Current term is 10.
I20260805 22:25:17.466005 31594 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:17.566066 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:17.567763 31444 leader_election.cc:401] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Cancelling election due to peer responding with higher term
I20260805 22:25:17.567993 32583 raft_consensus.cc:2379] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:17.564508 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:17.570185 32586 raft_consensus.cc:2379] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
W20260805 22:25:17.378703 3759 consensus_meta.cc:287] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.601s user 0.009s sys 0.001s
I20260805 22:25:17.422331 3736 raft_consensus.cc:3032] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 10 LEADER]: Stepping down as leader of term 10
I20260805 22:25:17.512130 3638 raft_consensus.cc:3037] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:17.579109 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:17.579861 3281 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:17.368394 3822 raft_consensus.cc:2781] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:25:17.580504 3822 raft_consensus.cc:686] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
W20260805 22:25:17.458534 3232 consensus_meta.cc:287] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.522s user 0.013s sys 0.003s
I20260805 22:25:17.581434 3759 raft_consensus.cc:2726] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 12 FOLLOWER]: Leader pre-election lost for term 11. Reason: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 11. Current term is 12.
I20260805 22:25:17.581825 3232 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:17.583850 3232 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:17.574278 32581 raft_consensus.cc:1261] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260805 22:25:17.571727 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:17.585741 31494 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:25:17.585991 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16e20d6ca2f34a65a5d7bc4471282feb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:17.586340 31494 raft_consensus.cc:595] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 9 LEADER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:17.586805 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:17.581351 3822 consensus_queue.cc:237] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 4.3, Last appended by leader: 3, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:17.585772 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:17.588879 32585 raft_consensus.cc:2379] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 8: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
W20260805 22:25:17.573014 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 60928.5us
I20260805 22:25:17.580003 32583 raft_consensus.cc:2379] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 8: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:17.575851 3736 raft_consensus.cc:729] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:17.573450 3592 consensus_queue.cc:237] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:17.590004 3780 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:17.587293 32529 leader_election.cc:334] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.579s (SENT)
I20260805 22:25:17.591353 32579 raft_consensus.cc:1226] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 4. Current term is 5. Ops: []
I20260805 22:25:17.592013 3780 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:17.592840 32529 leader_election.cc:334] T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.752s (SENT)
I20260805 22:25:17.593279 31604 raft_consensus.cc:1226] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260805 22:25:17.595479 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
W20260805 22:25:17.467087 3826 consensus_queue.cc:1159] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "516cb07ce3064ccbbca35e912d62816c" responder_term: 10 status { last_received { term: 10 index: 6 } last_committed_idx: 5 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 9. Current term is 10. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20260805 22:25:17.452337 3328 raft_consensus.cc:2726] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:17.596237 31604 raft_consensus.cc:2354] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6.
I20260805 22:25:17.596673 3705 consensus_queue.cc:1046] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:25:17.466740 3788 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:17.597687 3788 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:17.598289 3705 raft_consensus.cc:3032] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:25:17.598726 3705 raft_consensus.cc:729] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:17.596724 32529 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:17.599857 3788 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:17.428359 3746 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader election won for term 10
I20260805 22:25:17.599637 3705 consensus_queue.cc:260] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 5.7, Last appended by leader: 7, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:17.600449 3756 raft_consensus.cc:2726] T dcea14dac2934d139e2c5ab6d0b8648d P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:25:17.428447 3499 consensus_queue.cc:1035] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.178s
I20260805 22:25:17.601594 3746 raft_consensus.cc:686] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 10 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:17.604034 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:17.605053 31604 raft_consensus.cc:2379] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 7: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:17.525516 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:17.606386 32589 raft_consensus.cc:2379] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 4: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:17.607367 31707 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:17.607916 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:17.609757 31713 raft_consensus.cc:1226] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 8. Current term is 9. Ops: []
I20260805 22:25:17.600890 3705 raft_consensus.cc:3037] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:17.612118 32583 raft_consensus.cc:1261] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260805 22:25:17.614898 31444 leader_election.cc:398] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6.
I20260805 22:25:17.617672 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:17.617746 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:17.631024 31602 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:17.618803 31596 raft_consensus.cc:1226] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 10. Current term is 11. Ops: [8.6-10.7]
I20260805 22:25:17.626564 31708 raft_consensus.cc:2379] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 6: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:17.619819 3577 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 10 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:17.610304 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:17.621297 2855 consensus_queue.cc:1035] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260805 22:25:17.623373 3820 consensus_queue.cc:1046] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
I20260805 22:25:17.625114 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
W20260805 22:25:17.626485 3831 log.cc:924] Time spent T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.127s user 0.000s sys 0.001s
I20260805 22:25:17.633199 31444 leader_election.cc:401] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 election: Cancelling election due to peer responding with higher term
I20260805 22:25:17.633407 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:17.579078 3235 raft_consensus.cc:3037] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Advancing to term 10
I20260805 22:25:17.575941 32587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 5 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:17.618831 31594 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 9.
I20260805 22:25:17.422767 3493 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:17.736240 31602 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 7.
W20260805 22:25:17.748847 3828 log.cc:924] Time spent T 5e4ad7072aab4331878e988a26a6d887 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.182s user 0.000s sys 0.002s
W20260805 22:25:17.785827 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 171068us
I20260805 22:25:17.644856 3527 leader_election.cc:288] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:17.739078 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:17.428632 3800 consensus_queue.cc:1046] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.287s
I20260805 22:25:17.798133 3800 raft_consensus.cc:3032] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Stepping down as leader of term 7
I20260805 22:25:17.798517 3800 raft_consensus.cc:729] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:17.799328 3800 consensus_queue.cc:260] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:17.592768 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:17.719666 31711 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 10.
I20260805 22:25:17.690343 32529 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
W20260805 22:25:17.801685 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 111415us
I20260805 22:25:17.367134 3804 consensus_queue.cc:1046] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.206s
I20260805 22:25:17.803726 3804 raft_consensus.cc:3032] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:25:17.513465 31500 rpcz_store.cc:276] 0805 22:25:16.363174 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:16.363772 (+ 598us) service_pool.cc:227] Handling call
0805 22:25:16.364072 (+ 300us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:16.382046 (+ 17974us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:16.382072 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:16.389983 (+ 7911us) log.cc:842] Serialized 26 byte log entry
0805 22:25:16.771620 (+381637us) raft_consensus.cc:1631] Marking committed up to 2
0805 22:25:16.887063 (+115443us) log.cc:842] Serialized 14 byte log entry
0805 22:25:16.887259 (+ 196us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:16.887333 (+ 74us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:17.500422 (+613089us) raft_consensus.cc:1671] finished
0805 22:25:17.500443 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:17.500608 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18300,"prepare.queue_time_us":114705,"prepare.run_cpu_time_us":58,"prepare.run_wall_time_us":60,"raft.queue_time_us":50087,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":5916,"thread_start_us":338114,"threads_started":1,"wal-append.queue_time_us":566594}
I20260805 22:25:17.677850 3776 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:17.645290 3765 consensus_meta.cc:287] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385: Time spent flushing consensus metadata: real 0.655s user 0.007s sys 0.004s
I20260805 22:25:17.690881 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:17.805068 3765 raft_consensus.cc:504] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:17.692381 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:17.806277 31715 raft_consensus.cc:2379] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 11: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:17.806605 31708 raft_consensus.cc:2396] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260805 22:25:17.807096 3765 leader_election.cc:288] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 11 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:17.746716 31554 leader_election.cc:334] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.692s (SENT)
W20260805 22:25:17.739439 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 74977.9us
I20260805 22:25:17.807758 31554 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:17.808642 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 61964.3us
I20260805 22:25:17.810246 31554 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:17.675913 3771 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:17.710425 31705 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16"
dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
mode: GRACEFUL
new_leader_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:43958
W20260805 22:25:17.745800 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 447334us
I20260805 22:25:17.820724 31705 raft_consensus.cc:595] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 LEADER]: Received request to transfer leadership to TS 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:17.689908 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:17.821564 31601 raft_consensus.cc:2379] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 10: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:17.678247 3664 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:17.821895 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:17.680047 31556 leader_election.cc:302] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:17.822985 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 143019us
I20260805 22:25:17.823115 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:17.677286 3667 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:17.576660 2956 consensus_queue.cc:260] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:17.780920 32591 raft_consensus.cc:2396] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 8 because replica has last-logged OpId of term: 6 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 6.
I20260805 22:25:17.606887 3746 consensus_queue.cc:237] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:17.736934 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:17.724691 3577 consensus_queue.cc:1046] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 8.9, Next index: 10, Last known committed idx: 9, Time since last communication: 0.023s
I20260805 22:25:17.785224 32587 raft_consensus.cc:2379] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 5: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:17.731541 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:17.854717 31603 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 7.
I20260805 22:25:17.788291 31444 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
W20260805 22:25:17.855887 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 67660.2us
I20260805 22:25:17.788985 3232 consensus_queue.cc:1046] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 7.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:25:17.795794 3584 raft_consensus.cc:3037] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:17.803821 3572 raft_consensus.cc:2781] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election won for term 4
I20260805 22:25:17.807286 31715 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56256 (request call id 2052) took 1226 ms. Trace:
I20260805 22:25:17.867408 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:17.867656 31715 rpcz_store.cc:276] 0805 22:25:16.580970 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:16.581459 (+ 489us) service_pool.cc:227] Handling call
0805 22:25:17.807092 (+1225633us) spinlock_profiling.cc:243] Waited 1.08 s on lock 0x7b54001e2b54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:17.807228 (+ 136us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2488063744}
I20260805 22:25:17.814015 3788 raft_consensus.cc:2726] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:17.832181 3588 raft_consensus.cc:3037] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:17.814074 3765 consensus_queue.cc:1035] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.002s
I20260805 22:25:17.879583 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:17.880398 31714 raft_consensus.cc:2354] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 10. Current term is 11.
I20260805 22:25:17.687652 3803 consensus_queue.cc:237] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 5.3, Last appended by leader: 3, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:17.826212 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:17.881012 31708 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 7.
I20260805 22:25:17.823829 31713 raft_consensus.cc:2354] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 9. Current term is 10.
I20260805 22:25:17.831071 3768 raft_consensus.cc:2726] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:25:17.832551 31601 raft_consensus.cc:1226] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:17.834807 31711 raft_consensus.cc:1261] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 6 index: 6. (index mismatch)
I20260805 22:25:17.882839 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
W20260805 22:25:17.840693 31667 connection.cc:232] RPC call timeout handler was delayed by -1.4128222521394491s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.16810000000000003s for a response
I20260805 22:25:17.854509 3667 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:25:17.823894 31556 consensus_peers.cc:487] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c -> Peer 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801): Couldn't send request to peer 6bf8c694a0ba40978195ae044a917caf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07310455e35e48f8ba4eb9b72a69113b. This is attempt 1: this message will repeat every 5th retry.
I20260805 22:25:17.885038 31601 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
W20260805 22:25:17.888715 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 65101.8us
I20260805 22:25:17.868808 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:17.814560 3776 raft_consensus.cc:2781] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader election won for term 10
I20260805 22:25:17.873710 31609 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:17.889516 31556 leader_election.cc:302] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:17.874426 31444 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:17.875725 31447 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 299241us
I20260805 22:25:17.891260 3820 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:17.846760 32529 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:17.891458 3328 consensus_queue.cc:1035] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:25:17.891669 3297 consensus_queue.cc:1046] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 8.10, Next index: 11, Last known committed idx: 8, Time since last communication: 0.001s
W20260805 22:25:17.892373 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 45704.6us
I20260805 22:25:17.892880 3328 raft_consensus.cc:3032] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 8 LEADER]: Stepping down as leader of term 8
I20260805 22:25:17.893249 3328 raft_consensus.cc:729] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:17.894254 32529 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:17.894028 3328 consensus_queue.cc:260] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 8.10, Last appended by leader: 10, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:17.895248 3328 raft_consensus.cc:3037] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:17.884454 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:17.898017 31708 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:17.886503 31596 raft_consensus.cc:1226] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 6. Current term is 7. Ops: []
I20260805 22:25:17.867050 3690 raft_consensus.cc:2781] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:25:17.889653 31601 raft_consensus.cc:2379] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 11: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:17.804124 3804 raft_consensus.cc:729] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:17.895684 32594 raft_consensus.cc:1226] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 10 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 9. Current term is 10. Ops: []
I20260805 22:25:17.895690 31715 raft_consensus.cc:2354] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 10. Current term is 11.
I20260805 22:25:17.899384 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:17.920784 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:17.921265 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:17.922070 31711 raft_consensus.cc:2362] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 9. Re-sending same reply.
I20260805 22:25:17.923499 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:17.924521 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:17.925597 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:17.895196 3756 raft_consensus.cc:2726] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
I20260805 22:25:17.933943 3804 consensus_queue.cc:260] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:17.935635 3804 raft_consensus.cc:3037] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:17.937970 32594 raft_consensus.cc:1226] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
I20260805 22:25:17.938459 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:17.939543 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:17.940598 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:17.942103 32587 raft_consensus.cc:1226] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 7. Current term is 8. Ops: []
I20260805 22:25:17.925243 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:17.882906 3572 raft_consensus.cc:686] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:17.944406 3572 consensus_queue.cc:237] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:17.886677 3790 leader_election.cc:288] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:17.952998 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 45809.6us
W20260805 22:25:17.954640 31554 leader_election.cc:334] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.997s (SENT)
I20260805 22:25:17.955060 31554 leader_election.cc:302] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:17.942116 31713 raft_consensus.cc:1261] T 10de517cc0a24f20b5885091e99e252d P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260805 22:25:17.959615 32591 raft_consensus.cc:1226] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:17.948478 31715 raft_consensus.cc:1226] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 LEADER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 7. Current term is 8. Ops: []
I20260805 22:25:17.879890 3651 raft_consensus.cc:482] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:17.964774 31609 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 11.
I20260805 22:25:17.966570 31713 raft_consensus.cc:1261] T c46ca6ab004a4f2c90ecccced3235cb6 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
I20260805 22:25:17.969810 3690 raft_consensus.cc:686] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:17.971328 3690 consensus_queue.cc:237] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 6.5, Last appended by leader: 5, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:17.974257 31711 raft_consensus.cc:1226] T a260d4eadd0a4677aed088f331367c05 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 10. Current term is 11. Ops: [8.6-10.7]
I20260805 22:25:17.974524 31556 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:17.972323 31708 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 10.
I20260805 22:25:17.998574 32594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
W20260805 22:25:18.019778 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 135893us
I20260805 22:25:17.973259 3826 raft_consensus.cc:482] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:17.950347 3605 raft_consensus.cc:2781] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election won for term 8
W20260805 22:25:18.020854 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 126667us
I20260805 22:25:18.023890 31556 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:17.904493 3667 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:18.051957 31667 connection.cc:232] RPC call timeout handler was delayed by -1.3363222521729767s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1766s for a response
W20260805 22:25:18.065408 31667 connection.cc:232] RPC call timeout handler was delayed by -1.1959222520235926s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1922s for a response
W20260805 22:25:18.065729 31667 connection.cc:232] RPC call timeout handler was delayed by -1.143722252221778s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.198s for a response
I20260805 22:25:18.033049 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:17.982026 3664 raft_consensus.cc:2726] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:18.095449 31711 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
W20260805 22:25:18.098611 31667 connection.cc:232] RPC call timeout handler was delayed by -1.0024222522042692s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.2137s for a response
W20260805 22:25:18.078446 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 54670us
W20260805 22:25:18.103628 31667 leader_election.cc:334] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.634s (SENT)
I20260805 22:25:18.103914 31556 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 5 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:18.107476 31609 raft_consensus.cc:1261] T c46ca6ab004a4f2c90ecccced3235cb6 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 7 index: 5. Preceding OpId from leader: term: 9 index: 6. (index mismatch)
W20260805 22:25:18.060081 3842 log.cc:924] Time spent T e6cf90ae3fc840ceb48be8c790be97d9 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.291s user 0.001s sys 0.000s
I20260805 22:25:17.981397 3651 raft_consensus.cc:504] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:18.154068 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 156435us
I20260805 22:25:18.022805 3697 leader_election.cc:288] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:18.028465 3684 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:18.024164 3820 raft_consensus.cc:482] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:18.058679 3838 raft_consensus.cc:482] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:18.025072 3759 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:18.162211 31711 raft_consensus.cc:2454] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 7.
I20260805 22:25:18.062922 3776 raft_consensus.cc:686] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 10 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:18.189774 32529 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:18.095326 3605 raft_consensus.cc:482] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:18.170567 31554 leader_election.cc:302] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:18.206287 31554 leader_election.cc:302] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:18.174880 3684 raft_consensus.cc:2726] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election lost for term 5. Reason: could not achieve majority
W20260805 22:25:18.157707 3829 log.cc:924] Time spent T 16e20d6ca2f34a65a5d7bc4471282feb P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.131s user 0.001s sys 0.000s
I20260805 22:25:18.169512 3499 raft_consensus.cc:2726] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:17.997773 3760 consensus_queue.cc:1046] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.165s
I20260805 22:25:18.192057 31667 leader_election.cc:302] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:18.066440 3826 raft_consensus.cc:504] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:18.231431 32587 raft_consensus.cc:1226] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [term 4 LEADER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 3. Current term is 4. Ops: []
W20260805 22:25:18.235479 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 43506.1us
I20260805 22:25:18.207597 3605 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:18.208369 3697 raft_consensus.cc:2781] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:25:18.252686 3768 raft_consensus.cc:2726] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:25:18.253482 3867 raft_consensus.cc:2781] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election won for term 6
I20260805 22:25:18.244658 2452 raft_consensus.cc:2726] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:18.255267 3827 raft_consensus.cc:482] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:18.241510 3776 consensus_queue.cc:237] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:18.260385 3736 consensus_queue.cc:260] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:18.262310 3839 log.cc:924] Time spent T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.084s user 0.000s sys 0.000s
I20260805 22:25:18.265911 3771 raft_consensus.cc:3032] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 8 LEADER]: Stepping down as leader of term 8
W20260805 22:25:18.266348 3850 log.cc:924] Time spent T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.097s user 0.000s sys 0.000s
I20260805 22:25:18.269886 3868 raft_consensus.cc:482] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:18.270747 3826 leader_election.cc:288] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:18.274889 3232 raft_consensus.cc:482] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:18.201211 3838 raft_consensus.cc:504] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:18.188526 3820 raft_consensus.cc:504] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:18.203383 3759 raft_consensus.cc:504] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:18.295506 29783 auto_leader_rebalancer.cc:388] table: test-workload, leader rebalance finish, leader transfer count: 5
I20260805 22:25:18.430447 31601 raft_consensus.cc:1226] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 10. Current term is 11. Ops: [8.6-10.7]
W20260805 22:25:18.450475 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 167357us
I20260805 22:25:18.373869 31609 raft_consensus.cc:1226] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 9. Current term is 10. Ops: [9.9-9.9]
I20260805 22:25:18.455497 29783 auto_leader_rebalancer.cc:450] All tables' leader rebalancing finished this round
W20260805 22:25:18.319554 31556 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 215790us
I20260805 22:25:18.234201 3493 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:18.306183 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 52176.4us
I20260805 22:25:18.458010 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
W20260805 22:25:18.450224 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 244002us
I20260805 22:25:18.460940 31667 leader_election.cc:302] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:18.277398 3667 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:18.276075 3705 consensus_meta.cc:287] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf: Time spent flushing consensus metadata: real 0.648s user 0.010s sys 0.000s
W20260805 22:25:18.294565 3833 log.cc:924] Time spent T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.544s user 0.001s sys 0.001s
W20260805 22:25:18.454032 3804 consensus_meta.cc:287] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.518s user 0.013s sys 0.002s
W20260805 22:25:18.454170 3235 consensus_meta.cc:287] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.666s user 0.010s sys 0.003s
I20260805 22:25:18.276352 31357 rpcz_store.cc:275] Call kudu.master.MasterService.TSHeartbeat from 127.0.0.1:47946 (request call id 187) took 9633 ms. Trace:
I20260805 22:25:18.482378 31601 raft_consensus.cc:1261] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:25:18.531841 31705 raft_consensus.cc:1226] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 8. Current term is 9. Ops: []
I20260805 22:25:18.514114 32583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:18.532889 32583 raft_consensus.cc:2379] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 4: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:18.532902 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:18.524804 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:18.528180 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:18.452559 3771 raft_consensus.cc:729] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:18.493762 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:18.533663 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:18.544845 32585 raft_consensus.cc:2379] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 8: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:18.544343 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
W20260805 22:25:18.538121 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 241939us
I20260805 22:25:18.539521 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:18.279017 3651 leader_election.cc:288] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:18.416849 3584 consensus_meta.cc:287] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.547s user 0.004s sys 0.008s
I20260805 22:25:18.543577 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:18.546700 31444 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:18.482004 3736 raft_consensus.cc:3037] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:18.558405 32589 raft_consensus.cc:2379] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 10: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:18.549017 32581 raft_consensus.cc:2379] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 9: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
W20260805 22:25:18.547353 3850 log.cc:924] Time spent T 16e20d6ca2f34a65a5d7bc4471282feb P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.073s user 0.001s sys 0.000s
I20260805 22:25:18.477249 3577 raft_consensus.cc:482] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:18.428582 3818 raft_consensus.cc:3032] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 9 LEADER]: Stepping down as leader of term 9
I20260805 22:25:18.284772 3345 consensus_queue.cc:1046] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 9.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.251s
I20260805 22:25:18.555325 31708 raft_consensus.cc:2379] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 9: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:18.285647 3792 consensus_queue.cc:1046] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.253s
I20260805 22:25:18.473737 3867 raft_consensus.cc:686] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [term 6 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:18.556313 31444 leader_election.cc:398] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 10. Current term is 11.
I20260805 22:25:18.575428 31444 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:18.575592 31556 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:18.577693 31444 leader_election.cc:398] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 10. Current term is 11.
I20260805 22:25:18.578456 31556 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:18.578560 31444 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:18.531903 3490 raft_consensus.cc:2781] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Leader election won for term 11
I20260805 22:25:18.382222 3697 raft_consensus.cc:686] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 7 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:18.551502 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4172338662524cb29310f48e0861873e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:18.548807 32591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:18.537755 3328 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 12 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:18.284740 3621 consensus_queue.cc:1035] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.258s
I20260805 22:25:18.534173 3786 raft_consensus.cc:2726] T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:18.534785 3235 raft_consensus.cc:2726] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader pre-election lost for term 9. Reason: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 9. Current term is 10.
I20260805 22:25:18.455622 3232 raft_consensus.cc:504] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:18.472198 3827 raft_consensus.cc:504] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:18.562276 3826 raft_consensus.cc:2726] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260805 22:25:18.567538 3818 raft_consensus.cc:729] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:18.557062 3804 consensus_queue.cc:1035] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.030s
I20260805 22:25:18.521293 3883 consensus_queue.cc:1046] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 6.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.104s
I20260805 22:25:18.471987 3868 raft_consensus.cc:504] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:18.608420 3588 consensus_meta.cc:287] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.725s user 0.005s sys 0.007s
W20260805 22:25:18.599315 3866 log.cc:924] Time spent T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.174s user 0.000s sys 0.001s
I20260805 22:25:18.589821 3881 raft_consensus.cc:2726] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
I20260805 22:25:18.589762 3877 raft_consensus.cc:2726] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:25:18.570618 3577 raft_consensus.cc:504] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:18.587615 3584 raft_consensus.cc:2726] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader election lost for term 10. Reason: could not achieve majority
I20260805 22:25:18.645601 3818 consensus_queue.cc:260] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:18.590955 3490 raft_consensus.cc:686] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 11 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:18.599763 3650 raft_consensus.cc:2726] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election lost for term 8. Reason: could not achieve majority
W20260805 22:25:18.627133 3197 consensus_meta.cc:287] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 1.106s user 0.014s sys 0.000s
I20260805 22:25:18.637179 3820 leader_election.cc:288] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:18.648064 3854 log.cc:924] Time spent T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.108s user 0.000s sys 0.001s
I20260805 22:25:18.621544 3867 consensus_queue.cc:237] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.6, Last appended by leader: 6, Current term: 6, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:18.643720 3577 leader_election.cc:288] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:18.644958 3759 leader_election.cc:288] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:18.594521 3328 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 12 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:18.592785 3771 consensus_queue.cc:260] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:18.646941 3818 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:18.633505 3588 raft_consensus.cc:2726] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election lost for term 5. Reason: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6.
I20260805 22:25:18.670166 3868 leader_election.cc:288] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:18.676533 3904 ts_tablet_manager.cc:927] T 07310455e35e48f8ba4eb9b72a69113b P 6bf8c694a0ba40978195ae044a917caf: Initiating tablet copy from peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:18.529193 31357 rpcz_store.cc:276] 0805 22:25:08.644687 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:08.654822 (+ 10135us) service_pool.cc:227] Handling call
0805 22:25:18.276278 (+9621456us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0805 22:25:15.978074 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0805 22:25:15.978598 (+ 524us) write_op.cc:432] Acquiring schema lock in shared mode
0805 22:25:15.978637 (+ 39us) write_op.cc:435] Acquired schema lock
0805 22:25:15.978654 (+ 17us) tablet.cc:660] Decoding operations
0805 22:25:15.979895 (+ 1241us) write_op.cc:620] Acquiring the partition lock for write op
0805 22:25:15.979984 (+ 89us) write_op.cc:641] Partition lock acquired for write op
0805 22:25:15.980013 (+ 29us) tablet.cc:683] Acquiring locks for 14 operations
0805 22:25:15.981141 (+ 1128us) tablet.cc:699] Row locks acquired
0805 22:25:15.981157 (+ 16us) write_op.cc:260] PREPARE: finished
0805 22:25:15.981327 (+ 170us) write_op.cc:270] Start()
0805 22:25:16.036749 (+ 55422us) write_op.cc:276] Timestamp: P: 1785968715981270 usec, L: 0
0805 22:25:16.040114 (+ 3365us) op_driver.cc:331] REPLICATION: starting
0805 22:25:16.125867 (+ 85753us) log.cc:842] Serialized 5579 byte log entry
0805 22:25:16.949835 (+823968us) op_driver.cc:444] REPLICATION: finished
0805 22:25:17.089178 (+139343us) write_op.cc:301] APPLY: starting
0805 22:25:17.094937 (+ 5759us) tablet.cc:1365] starting BulkCheckPresence
0805 22:25:17.173019 (+ 78082us) tablet.cc:1368] finished BulkCheckPresence
0805 22:25:17.173164 (+ 145us) tablet.cc:1370] starting ApplyRowOperation cycle
0805 22:25:17.287104 (+113940us) tablet.cc:1381] finished ApplyRowOperation cycle
0805 22:25:17.296649 (+ 9545us) tablet_metrics.cc:531] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0805 22:25:17.301255 (+ 4606us) write_op.cc:312] APPLY: finished
0805 22:25:17.304546 (+ 3291us) log.cc:842] Serialized 101 byte log entry
0805 22:25:17.304707 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:25:17.304852 (+ 145us) write_op.cc:454] Released schema lock
0805 22:25:17.304901 (+ 49us) write_op.cc:341] FINISH: Updating metrics
0805 22:25:17.305195 (+ 294us) write_op.cc:489] Releasing partition, row and schema locks
0805 22:25:17.305224 (+ 29us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":966395,"reported_tablets":18,"child_traces":[["op",{"apply.queue_time_us":126988,"apply.run_cpu_time_us":4877,"apply.run_wall_time_us":226874,"num_ops":14,"prepare.queue_time_us":624473,"prepare.run_cpu_time_us":48693,"prepare.run_wall_time_us":404154,"raft.queue_time_us":149137,"raft.run_cpu_time_us":17421,"raft.run_wall_time_us":34964,"replication_time_us":903513,"thread_start_us":997793,"threads_started":4,"wal-append.queue_time_us":285738,"wal-append.run_cpu_time_us":20929,"wal-append.run_wall_time_us":1947155}]]}
I20260805 22:25:18.583880 31444 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:18.721417 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 137496us
W20260805 22:25:18.734558 31667 leader_election.cc:334] T 5df9cd03cde746e0956c8c7726d4eb70 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.672s (SENT)
I20260805 22:25:18.739814 31594 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56384 (request call id 464) took 1398 ms. Trace:
I20260805 22:25:18.740303 31596 raft_consensus.cc:1226] T 8513af2d574d45b4a6d02428543b1402 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 8. Current term is 9. Ops: []
I20260805 22:25:18.647847 3197 raft_consensus.cc:2726] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Leader election lost for term 10. Reason: Vote denied by peer 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897) with higher term. Message: Invalid argument: T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 10. Current term is 11.
I20260805 22:25:18.744120 3197 raft_consensus.cc:2726] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
I20260805 22:25:18.740361 31594 rpcz_store.cc:276] 0805 22:25:17.340926 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:17.341038 (+ 112us) service_pool.cc:227] Handling call
0805 22:25:18.739269 (+1398231us) spinlock_profiling.cc:243] Waited 105 ms on lock 0x7b540005d7d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:18.739552 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":241160576}
I20260805 22:25:18.748252 3328 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 13 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:18.750133 3232 leader_election.cc:288] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:18.658087 3490 consensus_queue.cc:237] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 9.6, Last appended by leader: 6, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:18.664151 3605 raft_consensus.cc:504] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:18.757014 3605 leader_election.cc:288] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:18.767446 31447 thread.cc:636] raft [worker] (thread pool) Time spent creating pthread: real 0.512s user 0.005s sys 0.001s
W20260805 22:25:18.768086 31447 thread.cc:602] raft [worker] (thread pool) Time spent starting thread: real 0.558s user 0.006s sys 0.001s
I20260805 22:25:18.771453 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:18.632161 3697 consensus_queue.cc:237] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:18.776860 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
W20260805 22:25:18.737289 31667 leader_election.cc:334] T cd72177fb2f24347ba19da011ccd3fe4 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.698s (SENT)
I20260805 22:25:18.667261 3771 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Advancing to term 9
W20260805 22:25:18.666970 3705 consensus_queue.cc:1159] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "feb5e7ab4b1541beaf2674c3c5500385" responder_term: 9 status { last_received { term: 8 index: 9 } last_committed_idx: 9 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 8. Current term is 9. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260805 22:25:18.781093 31667 leader_election.cc:334] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.784s (SENT)
I20260805 22:25:18.784734 31667 leader_election.cc:302] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:18.786350 3827 leader_election.cc:288] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:18.786605 31490 raft_consensus.cc:1226] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:18.787149 3780 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:18.788524 31490 raft_consensus.cc:1226] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Rejecting Update request from peer 6bf8c694a0ba40978195ae044a917caf for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:18.788179 3780 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:18.804939 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.805505 31600 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56384 (request call id 472) took 1440 ms. Trace:
I20260805 22:25:18.806111 31490 raft_consensus.cc:2454] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 8.
I20260805 22:25:18.807536 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.809810 3910 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:18.810770 3910 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:18.814811 31444 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:18.809039 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.784791 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:18.806123 31600 rpcz_store.cc:276] 0805 22:25:17.365285 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:17.366302 (+ 1017us) service_pool.cc:227] Handling call
0805 22:25:17.366577 (+ 275us) raft_consensus.cc:1457] Updating replica for 0 ops
0805 22:25:17.847395 (+480818us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:17.847426 (+ 31us) raft_consensus.cc:1525] Triggering prepare for 0 ops
0805 22:25:17.847471 (+ 45us) raft_consensus.cc:1631] Marking committed up to 9
0805 22:25:17.847500 (+ 29us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:18.804809 (+957309us) spinlock_profiling.cc:243] Waited 481 ms on lock 0x7b54003bf6d4. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f2731068f51 00007f273106799e 00007f2735fa92e9 00007f272f68f684 00007f272f68f612 00007f272f68f541 00007f272f68f4bc 00007f272f68e772 00007f272f49294c 00007f272f491d86
0805 22:25:18.804891 (+ 82us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:18.805232 (+ 341us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1105322240}
W20260805 22:25:18.820760 31667 leader_election.cc:334] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.918s (SENT)
I20260805 22:25:18.821431 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:18.821296 31667 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:18.821789 3909 raft_consensus.cc:2726] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260805 22:25:18.825312 3900 raft_consensus.cc:2726] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:18.826877 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:18.827839 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:18.805302 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:18.829044 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:18.820611 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:18.830518 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.831512 31502 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
I20260805 22:25:18.818449 3605 raft_consensus.cc:482] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:18.832610 3605 raft_consensus.cc:504] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:18.835054 3605 leader_election.cc:288] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:18.837720 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:18.838757 31709 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:56260 (request call id 2456) took 1012 ms. Trace:
I20260805 22:25:18.839151 31709 rpcz_store.cc:276] 0805 22:25:17.826414 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:17.835363 (+ 8949us) service_pool.cc:227] Handling call
0805 22:25:18.837440 (+1002077us) spinlock_profiling.cc:243] Waited 1.41 ms on lock 0x7b58002d0698. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e108851 00007f272e108bce 000000000034aea9 00007f2735f34a30 00007f2735ffa7b4 00007f2735ffa821 00007f2735faa1b0 00007f2735fa9a94 00007f272f691c04 00007f272f691b92
0805 22:25:18.838667 (+ 1227us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3246848}
I20260805 22:25:18.846418 3684 raft_consensus.cc:3032] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:25:18.847060 3684 raft_consensus.cc:729] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:18.847391 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.848949 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.849435 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.829540 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.849812 31500 raft_consensus.cc:2454] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 7.
I20260805 22:25:18.864277 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.864960 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.868651 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.873920 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.875101 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0d26905694943b7135a9b805cf106" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.866175 3827 raft_consensus.cc:2726] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:25:18.878284 31499 raft_consensus.cc:2454] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
I20260805 22:25:18.873178 3910 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:18.881232 31490 raft_consensus.cc:2454] T f914c8243e724eda849ce1fc2dc30827 P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 2.
W20260805 22:25:18.882520 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 68068.3us
I20260805 22:25:18.885207 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:18.886554 3827 consensus_queue.cc:1035] T 10de517cc0a24f20b5885091e99e252d P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:25:18.887471 3684 consensus_queue.cc:260] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:18.892709 3844 raft_consensus.cc:3032] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:25:18.886364 31600 raft_consensus.cc:1261] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260805 22:25:18.893311 3844 raft_consensus.cc:729] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:18.896608 3684 raft_consensus.cc:3037] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Advancing to term 5
I20260805 22:25:18.897910 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.894871 3844 consensus_queue.cc:260] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:18.899572 3844 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:18.897927 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d68a856424b4ceb8f4f54b4050e8dc4" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.905521 31493 raft_consensus.cc:2396] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260805 22:25:18.905629 31500 raft_consensus.cc:2454] T 94d0d26905694943b7135a9b805cf106 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 8.
W20260805 22:25:18.906121 3690 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.664s user 0.002s sys 0.009s
I20260805 22:25:18.910140 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.911535 31492 raft_consensus.cc:2379] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6bf8c694a0ba40978195ae044a917caf in current term 5: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:18.911434 3297 consensus_queue.cc:1046] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 7.8, Next index: 9, Last known committed idx: 8, Time since last communication: 0.001s
W20260805 22:25:18.912458 3690 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.708s user 0.002s sys 0.009s
W20260805 22:25:18.913026 3874 log.cc:924] Time spent T 16e20d6ca2f34a65a5d7bc4471282feb P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.664s user 0.000s sys 0.004s
W20260805 22:25:18.916109 3846 log.cc:924] Time spent T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.663s user 0.005s sys 0.000s
I20260805 22:25:18.916239 31597 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:59030 (request call id 1457) took 1086 ms. Trace:
I20260805 22:25:18.918838 31704 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:56256 (request call id 2076) took 1175 ms. Trace:
I20260805 22:25:18.919235 31704 rpcz_store.cc:276] 0805 22:25:17.743321 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:17.746292 (+ 2971us) service_pool.cc:227] Handling call
0805 22:25:17.779688 (+ 33396us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:17.780044 (+ 356us) raft_consensus.cc:1520] Early marking committed up to index 0
0805 22:25:17.780070 (+ 26us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:17.785113 (+ 5043us) log.cc:842] Serialized 506 byte log entry
0805 22:25:17.963489 (+178376us) raft_consensus.cc:1631] Marking committed up to 3
0805 22:25:17.963526 (+ 37us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:17.963590 (+ 64us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:18.900450 (+936860us) spinlock_profiling.cc:243] Waited 35.6 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:25:18.918354 (+ 17904us) raft_consensus.cc:1671] finished
0805 22:25:18.918377 (+ 23us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:18.918651 (+ 274us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":81882368,"thread_start_us":174527,"threads_started":1,"wal-append.queue_time_us":389900}
W20260805 22:25:18.925916 3875 log.cc:924] Time spent T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.673s user 0.000s sys 0.002s
I20260805 22:25:18.929442 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.946276 31490 raft_consensus.cc:2454] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 4.
I20260805 22:25:18.947419 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:18.948782 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:18.953363 31500 raft_consensus.cc:2354] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 5. Current term is 6.
I20260805 22:25:18.959435 3800 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:18.960286 3838 leader_election.cc:288] T c75e772a0f924080865ddf85ed1e2d86 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:18.912472 3297 consensus_queue.cc:1225] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer feb5e7ab4b1541beaf2674c3c5500385 log is divergent from this leader: its last log entry 8.9 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 8
I20260805 22:25:18.951483 31600 raft_consensus.cc:1261] T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 5 index: 7. (index mismatch)
I20260805 22:25:18.962927 3690 raft_consensus.cc:3032] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Stepping down as leader of term 7
I20260805 22:25:18.963809 3690 raft_consensus.cc:729] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:18.964257 31608 raft_consensus.cc:1261] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260805 22:25:18.964759 31596 raft_consensus.cc:1261] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 12 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 8 index: 6. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260805 22:25:18.963425 31594 raft_consensus.cc:1261] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 7 index: 8. (index mismatch)
I20260805 22:25:18.966001 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:18.966823 3690 consensus_queue.cc:260] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:18.916622 31597 rpcz_store.cc:276] 0805 22:25:17.829983 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:17.830912 (+ 929us) service_pool.cc:227] Handling call
0805 22:25:17.831191 (+ 279us) raft_consensus.cc:1457] Updating replica for 1 ops
0805 22:25:17.831503 (+ 312us) raft_consensus.cc:1520] Early marking committed up to index 7
0805 22:25:18.140548 (+309045us) log.cc:842] Serialized 14 byte log entry
0805 22:25:18.192667 (+ 52119us) log.cc:842] Serialized 14 byte log entry
0805 22:25:18.192753 (+ 86us) raft_consensus.cc:1525] Triggering prepare for 1 ops
0805 22:25:18.193194 (+ 441us) log.cc:842] Serialized 26 byte log entry
0805 22:25:18.193300 (+ 106us) raft_consensus.cc:1631] Marking committed up to 8
0805 22:25:18.194751 (+ 1451us) log.cc:842] Serialized 14 byte log entry
0805 22:25:18.255695 (+ 60944us) spinlock_profiling.cc:243] Waited 8.63 ms on lock 0x7b4800020c58. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2735f0f7f1 00007f2735f068e7 00007f2735f0677a 00007f2735ffa95f 00007f2735ffa627 00007f273600eb2c 00007f273600ead8 00007f273600ea58 00007f273600ea13 00007f273600dcbc 00007f2734f8fbe6
0805 22:25:18.530939 (+275244us) raft_consensus.cc:1683] Filling consensus response to leader.
0805 22:25:18.531097 (+ 158us) raft_consensus.cc:1654] Waiting on the replicates to finish logging
0805 22:25:18.755105 (+224008us) spinlock_profiling.cc:243] Waited 54.2 ms on lock 0x7b1800354f40. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2736fdf8b1 00007f2736fec661 00007f272e10df96 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2735f0f7f1 00007f2735f068e7 00007f2735f0677a 00007f2735ffa95f
0805 22:25:18.755572 (+ 467us) spinlock_profiling.cc:243] Waited 65.9 ms on lock 0x7b4800020c58. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2735f0f7f1 00007f2735f068e7 00007f2735f0677a 00007f2735ffa95f 00007f2735ffa627 00007f273600eb2c 00007f273600ead8 00007f273600ea58 00007f273600ea13 00007f273600dcbc 00007f2734f8fbe6
0805 22:25:18.912009 (+156437us) spinlock_profiling.cc:243] Waited 27.4 ms on lock 0x7b44000232cc. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f272df84f11 00007f272e052a95 00007f272e0534b2 00007f2730cfb511 00007f2730cf6e1a 00007f2730ceabee 00007f2730ce9f36 00007f2730ce9b58 00007f2730cf3551 00007f2730cf3509 00007f2730cf3499
0805 22:25:18.915823 (+ 3814us) raft_consensus.cc:1671] finished
0805 22:25:18.915844 (+ 21us) raft_consensus.cc:1677] UpdateReplicas() finished
0805 22:25:18.916072 (+ 228us) inbound_call.cc:173] Queueing success response
Metrics: {"prepare.queue_time_us":228429,"prepare.run_cpu_time_us":113,"prepare.run_wall_time_us":116,"raft.queue_time_us":113768,"raft.run_cpu_time_us":8247,"raft.run_wall_time_us":975208,"spinlock_wait_cycles":359027328,"thread_start_us":627268,"threads_started":3,"wal-append.queue_time_us":70568}
I20260805 22:25:18.969959 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.970763 31499 raft_consensus.cc:2454] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 6.
I20260805 22:25:18.971972 3904 tablet_copy_client.cc:316] T 07310455e35e48f8ba4eb9b72a69113b P 6bf8c694a0ba40978195ae044a917caf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.21.194:41897
W20260805 22:25:18.969087 31667 leader_election.cc:334] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.048s (SENT)
I20260805 22:25:18.973415 31667 leader_election.cc:302] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:18.974663 3844 raft_consensus.cc:2726] T c75e772a0f924080865ddf85ed1e2d86 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260805 22:25:18.975548 31667 leader_election.cc:334] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.105s (SENT)
W20260805 22:25:18.979323 3847 log.cc:924] Time spent T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.800s user 0.002s sys 0.000s
W20260805 22:25:18.981120 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 266169us
I20260805 22:25:18.987463 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.988299 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:18.988648 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:18.989774 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:18.990108 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:18.990933 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:18.993306 31495 raft_consensus.cc:2379] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 6: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:18.995609 31494 raft_consensus.cc:2454] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 9.
I20260805 22:25:18.996294 31500 raft_consensus.cc:2379] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 7: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:18.996508 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:18.996652 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:18.996796 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:18.997198 32586 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 9.
W20260805 22:25:18.996322 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 58230.6us
I20260805 22:25:18.998148 3592 consensus_queue.cc:1035] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:25:18.969179 3690 raft_consensus.cc:3037] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:19.001188 3297 raft_consensus.cc:482] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:19.002847 3297 raft_consensus.cc:504] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:19.007884 31502 raft_consensus.cc:2354] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for earlier term 9. Current term is 10.
I20260805 22:25:18.997449 3838 raft_consensus.cc:482] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:19.009852 3838 raft_consensus.cc:504] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:19.007790 31554 leader_election.cc:302] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:19.013708 31492 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 11.
I20260805 22:25:19.014768 3684 raft_consensus.cc:2726] T 4172338662524cb29310f48e0861873e P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:19.015139 31493 raft_consensus.cc:2379] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 8: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:19.016120 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 11 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:19.027992 3780 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:19.029500 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.030408 31495 raft_consensus.cc:2454] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
W20260805 22:25:19.002170 3919 log.cc:924] Time spent T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.107s user 0.000s sys 0.001s
I20260805 22:25:18.997120 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.033780 31500 raft_consensus.cc:2362] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 9. Re-sending same reply.
I20260805 22:25:19.038103 3838 leader_election.cc:288] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:19.039422 3827 consensus_queue.cc:1035] T c46ca6ab004a4f2c90ecccced3235cb6 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:25:19.040287 3896 raft_consensus.cc:3032] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:25:19.040772 3896 raft_consensus.cc:729] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:19.041759 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:19.042696 31492 raft_consensus.cc:2396] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260805 22:25:19.042032 3896 consensus_queue.cc:260] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.5, Last appended by leader: 5, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:19.043705 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:19.043890 3896 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:19.045667 3803 raft_consensus.cc:3032] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 LEADER]: Stepping down as leader of term 9
I20260805 22:25:19.046068 3803 raft_consensus.cc:729] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:19.047326 2956 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:19.047040 3803 consensus_queue.cc:260] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 9, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:19.049074 3803 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:19.033113 31351 catalog_manager.cc:5495] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 8 to 11, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 11 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:25:19.062909 31351 catalog_manager.cc:5495] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 8 to 9, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 9 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:19.064232 3297 leader_election.cc:288] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:19.064286 3765 raft_consensus.cc:979] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385: : Instructing follower 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:25:19.065593 3765 raft_consensus.cc:1067] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 LEADER]: Signalling peer 6bf8c694a0ba40978195ae044a917caf to start an election
I20260805 22:25:19.064926 31351 catalog_manager.cc:5495] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 7 to 9, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 9 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:19.074821 3505 consensus_queue.cc:1035] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20260805 22:25:19.084005 31554 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:19.085243 3869 raft_consensus.cc:2726] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:19.087836 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.088948 31500 raft_consensus.cc:2396] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 for term 8 because replica has last-logged OpId of term: 7 index: 4, which is greater than that of the candidate, which has last-logged OpId of term: 5 index: 3.
I20260805 22:25:19.085357 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "58a08ace2d684c14972d495f4a387a85" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 3 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.102872 31500 raft_consensus.cc:1226] T 9d68a856424b4ceb8f4f54b4050e8dc4 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 4. Current term is 5. Ops: []
I20260805 22:25:19.108896 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.112284 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:19.110150 3505 consensus_queue.cc:1035] T 4af8570c6c58453685d7a8f38b519905 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20260805 22:25:19.110675 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:19.115151 31500 raft_consensus.cc:2362] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 9. Re-sending same reply.
I20260805 22:25:19.116395 3765 consensus_queue.cc:1035] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.001s
I20260805 22:25:19.118494 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:19.119637 31597 raft_consensus.cc:1226] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 10. Current term is 11. Ops: []
I20260805 22:25:19.125813 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:19.126267 31554 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:19.127995 3684 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:19.129078 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:19.099253 3844 consensus_queue.cc:1035] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.010s
I20260805 22:25:19.133747 31499 raft_consensus.cc:2454] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
I20260805 22:25:19.145901 31714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07310455e35e48f8ba4eb9b72a69113b" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:25:19.148191 3664 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.894s user 0.001s sys 0.010s
W20260805 22:25:19.150411 3664 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.968s user 0.002s sys 0.011s
I20260805 22:25:19.130476 3776 raft_consensus.cc:2932] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [term 5 LEADER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:25:19.157289 31490 raft_consensus.cc:1226] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [term 7 LEADER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 6. Current term is 7. Ops: []
W20260805 22:25:19.180769 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 56859.6us
I20260805 22:25:19.140513 31351 catalog_manager.cc:5495] T 527d60ab6db74595bed7e87ca2ecc6e8 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 6 to 7, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 7 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: HEALTHY } } }
I20260805 22:25:19.175621 2452 consensus_queue.cc:1035] T 3f76fdd524184682ae27abc293a7fff6 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20260805 22:25:19.190501 31714 raft_consensus.cc:1226] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260805 22:25:19.195103 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:19.190678 3684 raft_consensus.cc:2726] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:19.191730 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:19.192118 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:25:19.193123 31667 leader_election.cc:334] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.659s (SENT)
I20260805 22:25:19.197222 31667 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:19.173266 31668 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:19.131227 3878 consensus_queue.cc:1035] T 49bb7cd9b51b4dd782a68b560d7570f2 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 4, Time since last communication: 0.043s
I20260805 22:25:19.199340 31501 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 8.
I20260805 22:25:19.196803 31502 raft_consensus.cc:2454] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 12 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 12.
I20260805 22:25:19.206872 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.207850 31709 raft_consensus.cc:2354] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 9. Current term is 10.
I20260805 22:25:19.207899 31502 raft_consensus.cc:2379] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 6: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:19.214869 31351 catalog_manager.cc:5495] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 4 to 7, leader changed from f8e9f4c756da4203917c464e2b179994 (127.29.21.193) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 7 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:19.216924 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:19.219338 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:19.221043 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:19.222270 3818 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:19.221765 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.222841 3818 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:19.224327 31495 raft_consensus.cc:2454] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:19.226246 31492 raft_consensus.cc:1261] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 12 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260805 22:25:19.402992 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:19.372605 32529 leader_election.cc:302] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:19.261657 31490 raft_consensus.cc:1261] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260805 22:25:19.405428 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:19.293812 31494 raft_consensus.cc:2454] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:25:19.410691 3818 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:19.411823 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:19.411972 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 218880us
I20260805 22:25:19.412690 3490 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:19.417796 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:19.419335 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:19.420099 31713 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 12.
I20260805 22:25:19.416535 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.422152 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:19.422704 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:19.423321 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:19.418076 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45cd56d4e96040f98dbf1ae001a1005a" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:19.418807 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:19.257663 31500 raft_consensus.cc:1261] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 9 index: 5. (index mismatch)
I20260805 22:25:19.418807 31714 raft_consensus.cc:1261] T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260805 22:25:19.423830 31495 raft_consensus.cc:2454] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 12 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 12.
W20260805 22:25:19.432806 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 60330.9us
W20260805 22:25:19.362651 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 219937us
W20260805 22:25:19.361913 31554 leader_election.cc:334] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.514s (SENT)
I20260805 22:25:19.441576 31502 raft_consensus.cc:1261] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 4 index: 3. Preceding OpId from leader: term: 6 index: 4. (index mismatch)
I20260805 22:25:19.451506 31554 leader_election.cc:302] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:19.373221 3803 raft_consensus.cc:2781] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260805 22:25:19.459136 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8abfbdc4afff482ebc81c4fda5d0812a" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.471642 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a1aa2dc5e5341f0a0450361d6c95814" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.475656 31500 raft_consensus.cc:2379] T 6a1aa2dc5e5341f0a0450361d6c95814 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 6: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:19.474598 31495 raft_consensus.cc:2454] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 2.
I20260805 22:25:19.482328 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "52af4d94f396434d997b46f52763cd20" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:25:19.472487 3935 log.cc:924] Time spent T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.080s user 0.000s sys 0.001s
I20260805 22:25:19.454975 3605 raft_consensus.cc:2781] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 9 FOLLOWER]: Leader election won for term 9
I20260805 22:25:19.490559 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b16a2e0814140e39609da87479bb57e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.501233 3857 raft_consensus.cc:482] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:19.501772 3869 raft_consensus.cc:482] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:19.501363 31500 raft_consensus.cc:2454] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6.
I20260805 22:25:19.503230 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5b0fd7f6d7a46b2a77e841ceda358ef" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.482045 3297 raft_consensus.cc:482] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:19.502079 3882 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:19.515663 31500 raft_consensus.cc:2454] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:19.505411 31502 raft_consensus.cc:2454] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
W20260805 22:25:19.520524 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 158687us
I20260805 22:25:19.517146 3869 raft_consensus.cc:504] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:19.517858 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 58039.1us
I20260805 22:25:19.521332 31707 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 8.
W20260805 22:25:19.514072 3929 log.cc:924] Time spent T e6cf90ae3fc840ceb48be8c790be97d9 P 6bf8c694a0ba40978195ae044a917caf: Append to log took a long time: real 0.113s user 0.001s sys 0.000s
I20260805 22:25:19.512199 31351 catalog_manager.cc:5495] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 7 to 9, leader changed from 516cb07ce3064ccbbca35e912d62816c (127.29.21.194) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 9 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } health_report { overall_health: UNKNOWN } } }
I20260805 22:25:19.511910 3605 raft_consensus.cc:686] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [term 9 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:19.543301 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.505141 3873 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:19.564597 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.481590 3803 raft_consensus.cc:482] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:19.575088 31709 raft_consensus.cc:1261] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 12 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 11 index: 7. Preceding OpId from leader: term: 12 index: 8. (index mismatch)
I20260805 22:25:19.589089 31493 raft_consensus.cc:2379] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 9: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:19.588929 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:19.589891 31609 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
I20260805 22:25:19.513908 3857 raft_consensus.cc:504] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:19.593899 3297 raft_consensus.cc:504] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:19.594461 31704 raft_consensus.cc:1226] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [term 11 LEADER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 10. Current term is 11. Ops: []
I20260805 22:25:19.595309 3857 leader_election.cc:288] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:19.597584 3873 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:19.600042 3873 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:19.601706 32529 leader_election.cc:334] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.596s (SENT)
I20260805 22:25:19.602244 32529 leader_election.cc:302] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:19.603623 32529 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.773s (SENT) [suppressed 83 similar messages]
W20260805 22:25:19.605370 32529 leader_election.cc:334] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.636s (SENT)
I20260805 22:25:19.581071 31351 catalog_manager.cc:5495] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 6 to 8. New cstate: current_term: 8 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:19.593729 31500 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 10.
I20260805 22:25:19.625614 32529 leader_election.cc:302] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:19.582438 3803 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:19.569888 3844 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
W20260805 22:25:19.627907 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 62168.8us
I20260805 22:25:19.647053 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:19.653697 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.654583 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:19.658102 31500 raft_consensus.cc:2362] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 12 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 12. Re-sending same reply.
W20260805 22:25:19.651942 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 46594.6us
I20260805 22:25:19.630755 3906 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:25:19.632695 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 101098us
I20260805 22:25:19.622138 3857 raft_consensus.cc:2726] T 36ee93639c3f4084b94145c6ee60a2a8 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
W20260805 22:25:19.655220 31707 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:58348 (request call id 659) took 2048 ms (client timeout 1952 ms). Trace:
W20260805 22:25:19.669489 31707 rpcz_store.cc:269] 0805 22:25:17.606981 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:17.607120 (+ 139us) service_pool.cc:227] Handling call
0805 22:25:19.654960 (+2047840us) spinlock_profiling.cc:243] Waited 1.58 s on lock 0x7b5400152a54. stack: 00007f272e10e033 00007f272e10df30 00007f272da6c534 00007f27377788b4 00007f2737778849 00007f2730fed471 00007f273106c2ae 00007f2735fa9c27 00007f272f691c04 00007f272f691b92 00007f272f691ac1 00007f272f691a3c 00007f272f690cf2 00007f272f49294c 00007f272f491d86 00007f272f49172f
0805 22:25:19.655097 (+ 137us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3627215616}
I20260805 22:25:19.691943 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7c99e6a491454b5ba8d31e3246e77797" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:19.692948 31602 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:19.702307 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "986c28960d44411487f79ca134af59e2" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 7 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:19.704197 31609 raft_consensus.cc:1261] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260805 22:25:19.704514 31623 tablet_copy_service.cc:140] P 516cb07ce3064ccbbca35e912d62816c: Received BeginTabletCopySession request for tablet 07310455e35e48f8ba4eb9b72a69113b from peer 6bf8c694a0ba40978195ae044a917caf ({username='slave'} at 127.0.0.1:56384)
I20260805 22:25:19.705255 31623 tablet_copy_service.cc:161] P 516cb07ce3064ccbbca35e912d62816c: Beginning new tablet copy session on tablet 07310455e35e48f8ba4eb9b72a69113b from peer 6bf8c694a0ba40978195ae044a917caf at {username='slave'} at 127.0.0.1:56384: session id = 6bf8c694a0ba40978195ae044a917caf-07310455e35e48f8ba4eb9b72a69113b
I20260805 22:25:19.716341 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:19.717810 31709 raft_consensus.cc:2932] T 07310455e35e48f8ba4eb9b72a69113b P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Committing config change with OpId 5.4: config changed from index -1 to 4, NON_VOTER 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } attrs { replace: true } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: NON_VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: true } } }
I20260805 22:25:19.720542 31602 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 8.
I20260805 22:25:19.722254 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:19.723047 31554 leader_election.cc:334] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.553s (SENT)
I20260805 22:25:19.722919 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:19.578161 3882 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:19.738176 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 79062.9us
I20260805 22:25:19.746397 3771 raft_consensus.cc:2726] T f914c8243e724eda849ce1fc2dc30827 P 6bf8c694a0ba40978195ae044a917caf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:25:19.665336 3906 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:19.746994 3776 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.BulkChangeConfig from 127.0.0.1:36208 (request call id 11) took 2973 ms. Trace:
I20260805 22:25:19.747438 3776 rpcz_store.cc:276] 0805 22:25:16.773954 (+ 0us) service_pool.cc:170] Inserting onto call queue
0805 22:25:16.774286 (+ 332us) service_pool.cc:227] Handling call
0805 22:25:16.952231 (+177945us) log.cc:842] Serialized 506 byte log entry
0805 22:25:19.746870 (+2794639us) inbound_call.cc:173] Queueing success response
Metrics: {"dns_us":176,"mutex_wait_us":50430,"raft.queue_time_us":64619,"raft.run_cpu_time_us":1512,"raft.run_wall_time_us":1728,"thread_start_us":76478,"threads_started":1,"wal-append.queue_time_us":165605,"wal-append.run_cpu_time_us":5751,"wal-append.run_wall_time_us":1019973}
I20260805 22:25:19.653817 3605 consensus_queue.cc:237] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.10, Last appended by leader: 10, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:19.746426 31554 leader_election.cc:302] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
W20260805 22:25:19.754148 31554 leader_election.cc:334] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.562s (SENT)
I20260805 22:25:19.754642 31554 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:19.755970 31554 leader_election.cc:334] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.661s (SENT)
I20260805 22:25:19.756397 31554 leader_election.cc:302] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:19.756687 31704 raft_consensus.cc:3037] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:19.760462 31594 raft_consensus.cc:1261] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260805 22:25:19.761308 3882 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:19.762328 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:19.763037 31713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3f76fdd524184682ae27abc293a7fff6" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 5 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:19.764043 31715 raft_consensus.cc:1261] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Refusing update from remote peer 6bf8c694a0ba40978195ae044a917caf: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260805 22:25:19.765034 3684 raft_consensus.cc:3037] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Advancing to term 6
I20260805 22:25:19.767884 32579 raft_consensus.cc:1261] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 7 index: 8. Preceding OpId from leader: term: 8 index: 9. (index mismatch)
I20260805 22:25:19.769811 3882 raft_consensus.cc:2726] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
W20260805 22:25:19.770146 3951 log.cc:924] Time spent T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.133s user 0.001s sys 0.000s
I20260805 22:25:19.770491 3882 raft_consensus.cc:2726] T 4af8570c6c58453685d7a8f38b519905 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:19.771454 3800 consensus_queue.cc:1046] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: INVALID_TERM, Last received: 3.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
W20260805 22:25:19.772123 31554 leader_election.cc:334] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.686s (SENT)
I20260805 22:25:19.772544 3800 raft_consensus.cc:3032] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 6 LEADER]: Stepping down as leader of term 6
I20260805 22:25:19.772917 3800 raft_consensus.cc:729] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 6 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:19.773746 3800 consensus_queue.cc:260] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.6, Last appended by leader: 6, Current term: 6, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:19.774778 32586 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16"
dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
from {username='slave'} at 127.0.0.1:57366
I20260805 22:25:19.775197 3800 raft_consensus.cc:3037] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:19.776206 3684 raft_consensus.cc:2726] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:19.772585 31554 leader_election.cc:302] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:19.777987 3873 raft_consensus.cc:2726] T eda619941ce64f788d0b7a364eed0bdb P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:19.780515 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1115c6a30f64413ab24c9d215d58a2c7" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.781220 31494 raft_consensus.cc:2362] T 1115c6a30f64413ab24c9d215d58a2c7 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8. Re-sending same reply.
I20260805 22:25:19.781785 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.782281 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:19.783113 31493 raft_consensus.cc:2379] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 7: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:19.783754 31495 raft_consensus.cc:1226] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Rejecting Update request from peer feb5e7ab4b1541beaf2674c3c5500385 for earlier term 5. Current term is 6. Ops: [5.5-5.5]
W20260805 22:25:19.784552 31554 leader_election.cc:334] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.738s (SENT)
I20260805 22:25:19.785069 31554 leader_election.cc:302] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:19.785411 3937 log.cc:924] Time spent T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994: Append to log took a long time: real 0.116s user 0.000s sys 0.000s
I20260805 22:25:19.787496 3684 raft_consensus.cc:2726] T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:25:19.790112 3869 leader_election.cc:288] T dcea14dac2934d139e2c5ab6d0b8648d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:19.794226 3776 consensus_queue.cc:1035] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:25:19.796396 3297 leader_election.cc:288] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:19.650090 3844 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:19.798256 3776 consensus_queue.cc:1035] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260805 22:25:19.798803 3844 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:19.807648 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:19.808492 31706 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 10.
W20260805 22:25:19.809728 31667 leader_election.cc:334] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 3.195s (SENT)
I20260805 22:25:19.810289 31667 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:19.811434 31715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:25:19.813755 32529 leader_election.cc:334] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.640s (SENT)
I20260805 22:25:19.814248 32529 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:19.815428 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 77898.7us
I20260805 22:25:19.815982 31554 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:19.816794 3776 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260805 22:25:19.817168 3776 raft_consensus.cc:482] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:19.817512 3776 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:19.818002 32584 raft_consensus.cc:1261] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 5 index: 7. Preceding OpId from leader: term: 6 index: 7. (term mismatch)
I20260805 22:25:19.818312 32584 pending_rounds.cc:76] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf: Aborting all ops after (but not including) 6
I20260805 22:25:19.818610 32584 pending_rounds.cc:97] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf: Aborting uncommitted NO_OP operation due to leader change: 5.7
I20260805 22:25:19.818856 32584 raft_consensus.cc:2866] T 986c28960d44411487f79ca134af59e2 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:25:19.822857 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.783118 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8fd526bfd0047e49803dbf11f8eafc6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:19.782577 31490 raft_consensus.cc:2379] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 8: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:19.842793 31501 raft_consensus.cc:2379] T d8fd526bfd0047e49803dbf11f8eafc6 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 6: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
W20260805 22:25:19.828486 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 86461.9us
I20260805 22:25:19.849087 2452 raft_consensus.cc:2726] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260805 22:25:19.851284 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 7 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:19.852389 31500 raft_consensus.cc:1261] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 6 index: 4. Preceding OpId from leader: term: 8 index: 5. (index mismatch)
I20260805 22:25:19.862623 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:19.873041 31502 raft_consensus.cc:1261] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 7 index: 4. (index mismatch)
I20260805 22:25:19.863381 31603 raft_consensus.cc:2454] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:25:19.842474 3771 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:19.891316 31667 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:19.879514 31351 catalog_manager.cc:5495] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 4 to 7, leader changed from feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195) to 516cb07ce3064ccbbca35e912d62816c (127.29.21.194). New cstate: current_term: 7 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:19.884333 31444 leader_election.cc:398] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Vote denied by peer feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831) with higher term. Message: Invalid argument: T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 for earlier term 9. Current term is 10.
I20260805 22:25:19.900513 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 12 candidate_status { last_received { term: 10 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:19.901270 31502 raft_consensus.cc:2362] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 12 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 12. Re-sending same reply.
I20260805 22:25:19.857203 3776 raft_consensus.cc:504] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:19.922817 31554 leader_election.cc:334] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.703s (SENT)
I20260805 22:25:19.898154 3857 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:19.940114 3776 leader_election.cc:288] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 11 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:19.929981 31502 raft_consensus.cc:3032] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 10 LEADER]: Stepping down as leader of term 10
I20260805 22:25:19.944867 31502 raft_consensus.cc:729] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
W20260805 22:25:19.949944 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 105555us
I20260805 22:25:19.950989 31444 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 13 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:25:19.952057 31554 leader_election.cc:334] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.798s (SENT)
I20260805 22:25:19.952662 31554 leader_election.cc:302] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:19.960073 2452 raft_consensus.cc:2781] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260805 22:25:19.960544 2452 raft_consensus.cc:482] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:19.961109 2452 raft_consensus.cc:3037] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 4 FOLLOWER]: Advancing to term 5
W20260805 22:25:19.961613 32529 leader_election.cc:334] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.652s (SENT)
I20260805 22:25:19.963105 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:19.960820 3297 raft_consensus.cc:2781] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 12 FOLLOWER]: Leader pre-election won for term 13
I20260805 22:25:19.964376 3297 raft_consensus.cc:482] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 12 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260805 22:25:19.964599 3957 log.cc:924] Time spent T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260805 22:25:19.961182 3869 consensus_queue.cc:1035] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.004s
W20260805 22:25:19.962363 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 71334.9us
I20260805 22:25:19.970912 31704 raft_consensus.cc:2454] T 7c99e6a491454b5ba8d31e3246e77797 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 8.
I20260805 22:25:19.971889 32529 leader_election.cc:302] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:19.942709 3857 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:19.966919 3776 raft_consensus.cc:2726] T ab66683a9e5e46c2b09be4afdee2a08c P 516cb07ce3064ccbbca35e912d62816c [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260805 22:25:19.967195 3297 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 12 FOLLOWER]: Advancing to term 13
W20260805 22:25:19.978549 31423 auto_rebalancer.cc:254] failed to send replica move request: Illegal state: Replica 516cb07ce3064ccbbca35e912d62816c is not leader of this config. Role: FOLLOWER. Consensus state: current_term: 7 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:19.976859 31351 catalog_manager.cc:5495] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 2 to 4. New cstate: current_term: 4 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:19.981648 31351 catalog_manager.cc:5495] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 7 to 8, leader changed from 6bf8c694a0ba40978195ae044a917caf (127.29.21.196) to f8e9f4c756da4203917c464e2b179994 (127.29.21.193). New cstate: current_term: 8 leader_uuid: "f8e9f4c756da4203917c464e2b179994" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20260805 22:25:19.993777 31351 catalog_manager.cc:5495] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 8 to 10. New cstate: current_term: 10 leader_uuid: "feb5e7ab4b1541beaf2674c3c5500385" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:19.992442 31444 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:20.001426 31351 catalog_manager.cc:5495] T 52af4d94f396434d997b46f52763cd20 P f8e9f4c756da4203917c464e2b179994 reported cstate change: term changed from 3 to 5. New cstate: current_term: 5 leader_uuid: "516cb07ce3064ccbbca35e912d62816c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } }
I20260805 22:25:19.986522 3771 raft_consensus.cc:2726] T 94d0d26905694943b7135a9b805cf106 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260805 22:25:20.002067 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 68698us
W20260805 22:25:20.014322 31554 leader_election.cc:334] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.851s (SENT)
I20260805 22:25:20.004812 31502 consensus_queue.cc:260] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 10.7, Last appended by leader: 7, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:20.024452 31502 raft_consensus.cc:3037] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:20.027662 31499 raft_consensus.cc:2454] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 11.
W20260805 22:25:20.028753 3736 consensus_queue.cc:1159] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "feb5e7ab4b1541beaf2674c3c5500385" responder_term: 11 status { last_received { term: 11 index: 7 } last_committed_idx: 6 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 10. Current term is 11. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
W20260805 22:25:20.032585 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 74707.7us
I20260805 22:25:20.040287 3197 raft_consensus.cc:2726] T c5b0fd7f6d7a46b2a77e841ceda358ef P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:20.008548 3803 raft_consensus.cc:504] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:20.048601 32529 leader_election.cc:334] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.659s (SENT)
I20260805 22:25:20.042790 31444 leader_election.cc:302] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:20.054149 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:20.041301 3838 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:20.055032 32585 raft_consensus.cc:2379] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:20.059311 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c46ca6ab004a4f2c90ecccced3235cb6" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:20.059382 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:20.060245 31704 raft_consensus.cc:2362] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 8. Re-sending same reply.
W20260805 22:25:20.062678 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 48402.8us
I20260805 22:25:20.063267 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:20.064038 31490 raft_consensus.cc:2379] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 8: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:20.062913 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:20.066293 31492 raft_consensus.cc:2396] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260805 22:25:20.068212 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
W20260805 22:25:20.093130 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 50419.8us
I20260805 22:25:20.068871 31501 raft_consensus.cc:2362] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 6. Re-sending same reply.
I20260805 22:25:20.106592 31444 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:20.108470 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 64545.7us
I20260805 22:25:20.110543 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
W20260805 22:25:20.106249 3951 log.cc:924] Time spent T 32fee80a830a450195ce9c3462460ca1 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.051s user 0.000s sys 0.000s
I20260805 22:25:20.114145 31501 raft_consensus.cc:2379] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 11: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:20.104235 3197 raft_consensus.cc:2726] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:20.115124 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:20.121223 31708 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Advancing to term 11
W20260805 22:25:20.122138 32529 leader_election.cc:334] T 3095cdb25afc4c26bd9d1e03a919b474 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.662s (SENT)
W20260805 22:25:20.130787 31554 leader_election.cc:334] T cd72177fb2f24347ba19da011ccd3fe4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.103s (SENT)
I20260805 22:25:20.131722 31501 raft_consensus.cc:1226] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 8. Current term is 9. Ops: []
I20260805 22:25:20.061714 3838 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.140676 31444 leader_election.cc:302] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:20.143955 31444 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:20.148169 31444 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:20.111405 3765 consensus_queue.cc:1035] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.031s
W20260805 22:25:20.152537 31667 leader_election.cc:334] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.573s (SENT)
I20260805 22:25:20.162465 31667 leader_election.cc:302] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:20.165345 3786 raft_consensus.cc:2726] T 49bb7cd9b51b4dd782a68b560d7570f2 P 516cb07ce3064ccbbca35e912d62816c [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
I20260805 22:25:20.140230 3857 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:20.170217 3907 raft_consensus.cc:2726] T 45cd56d4e96040f98dbf1ae001a1005a P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:20.171124 3907 raft_consensus.cc:2726] T 8abfbdc4afff482ebc81c4fda5d0812a P f8e9f4c756da4203917c464e2b179994 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:25:20.152751 3818 raft_consensus.cc:2726] T 91a9acdbc98a4e799374fe6e8950c3a7 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:20.159550 3197 raft_consensus.cc:2726] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260805 22:25:20.180624 31554 leader_election.cc:334] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.130s (SENT)
I20260805 22:25:20.176877 31623 tablet_copy_source_session.cc:205] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c: Tablet Copy: opened 0 blocks and 1 log segments
I20260805 22:25:20.183336 31554 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:20.184818 31554 leader_election.cc:334] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.140s (SENT)
I20260805 22:25:20.185247 31554 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:20.186055 3873 raft_consensus.cc:2726] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260805 22:25:20.186692 3873 raft_consensus.cc:2726] T 5df9cd03cde746e0956c8c7726d4eb70 P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
W20260805 22:25:20.188336 31554 leader_election.cc:334] T 3095cdb25afc4c26bd9d1e03a919b474 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.201s (SENT)
W20260805 22:25:20.191140 31554 leader_election.cc:334] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.207s (SENT)
W20260805 22:25:20.194185 31554 leader_election.cc:334] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.207s (SENT)
W20260805 22:25:20.177250 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 55136.8us
I20260805 22:25:20.194989 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:20.206123 31501 raft_consensus.cc:2379] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 8: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:20.179497 3896 raft_consensus.cc:2726] T 58a08ace2d684c14972d495f4a387a85 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:25:20.212842 31502 raft_consensus.cc:1261] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 10 index: 7. Preceding OpId from leader: term: 11 index: 7. (term mismatch)
I20260805 22:25:20.214135 31502 pending_rounds.cc:76] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994: Aborting all ops after (but not including) 6
I20260805 22:25:20.214375 31502 pending_rounds.cc:97] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994: Aborting uncommitted NO_OP operation due to leader change: 10.7
I20260805 22:25:20.214602 31502 raft_consensus.cc:2866] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260805 22:25:20.215112 2452 raft_consensus.cc:504] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:20.198086 3960 log.cc:924] Time spent T 3f76fdd524184682ae27abc293a7fff6 P 516cb07ce3064ccbbca35e912d62816c: Append to log took a long time: real 0.088s user 0.001s sys 0.000s
I20260805 22:25:20.220522 2452 leader_election.cc:288] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:20.195386 3803 leader_election.cc:288] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:20.221648 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 78421.7us
I20260805 22:25:20.223032 32586 raft_consensus.cc:482] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Starting forced leader election (received explicit request)
I20260805 22:25:20.224371 32586 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:20.224392 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:20.229413 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:20.232115 31596 raft_consensus.cc:1261] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 9 index: 6. Preceding OpId from leader: term: 11 index: 7. (index mismatch)
W20260805 22:25:20.233361 32529 leader_election.cc:334] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.688s (SENT)
W20260805 22:25:20.237941 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 43972.8us
W20260805 22:25:20.239427 31667 leader_election.cc:334] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.574s (SENT)
W20260805 22:25:20.262373 31554 leader_election.cc:334] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.247s (SENT)
I20260805 22:25:20.255885 32586 raft_consensus.cc:504] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.276161 31554 leader_election.cc:302] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:20.277329 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49bb7cd9b51b4dd782a68b560d7570f2" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 12 candidate_status { last_received { term: 8 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:25:20.279241 31667 leader_election.cc:334] T 1115c6a30f64413ab24c9d215d58a2c7 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.578s (SENT)
W20260805 22:25:20.281704 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 63696.8us
I20260805 22:25:20.282516 31502 raft_consensus.cc:2379] T 49bb7cd9b51b4dd782a68b560d7570f2 P f8e9f4c756da4203917c464e2b179994 [term 12 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 12: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
W20260805 22:25:20.287318 31667 leader_election.cc:334] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.681s (SENT)
I20260805 22:25:20.291893 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:20.292616 31597 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:20.295404 31501 raft_consensus.cc:2454] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 10.
I20260805 22:25:20.297958 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
W20260805 22:25:20.303086 32529 leader_election.cc:334] T d8fd526bfd0047e49803dbf11f8eafc6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.733s (SENT)
I20260805 22:25:20.306372 31597 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 10.
I20260805 22:25:20.286518 3684 raft_consensus.cc:2726] T 10de517cc0a24f20b5885091e99e252d P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
W20260805 22:25:20.313449 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 165350us
I20260805 22:25:20.303612 31594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:20.320063 31444 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:20.324671 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.327659 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4af8570c6c58453685d7a8f38b519905" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.328404 31502 raft_consensus.cc:2379] T 4af8570c6c58453685d7a8f38b519905 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 8: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
W20260805 22:25:20.329341 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 222551us
W20260805 22:25:20.330370 31554 leader_election.cc:334] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.280s (SENT)
I20260805 22:25:20.317946 3838 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:20.324312 3776 raft_consensus.cc:482] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:20.333209 31554 leader_election.cc:302] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:20.333709 3776 raft_consensus.cc:504] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.335059 3973 raft_consensus.cc:2726] T 18a7e330200a49349edcae379f1e2b57 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
W20260805 22:25:20.335631 3975 log.cc:924] Time spent T 0ff925b5d86b41c19d809f76e57ab062 P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.058s user 0.000s sys 0.000s
I20260805 22:25:20.335986 3776 leader_election.cc:288] T d264e8379ab54b4785c836a00397ba95 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:20.337503 3978 log.cc:924] Time spent T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385: Append to log took a long time: real 0.081s user 0.001s sys 0.000s
W20260805 22:25:20.338671 3800 consensus_meta.cc:287] T 58a08ace2d684c14972d495f4a387a85 P f8e9f4c756da4203917c464e2b179994: Time spent flushing consensus metadata: real 0.563s user 0.012s sys 0.001s
I20260805 22:25:20.340137 3297 raft_consensus.cc:504] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 13 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:20.340982 31423 auto_rebalancer.cc:663] Could not move replica: Incomplete: tablet 89a7aa19faed455fb2600ae6851aee0e, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:20.340345 3973 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:20.340337 3788 raft_consensus.cc:482] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260805 22:25:20.341353 31423 auto_rebalancer.cc:264] scheduled replica move failed to complete: Incomplete: tablet 89a7aa19faed455fb2600ae6851aee0e, TS f8e9f4c756da4203917c464e2b179994 -> TS 6bf8c694a0ba40978195ae044a917caf move failed, destination replica disappeared from tablet's Raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:20.341634 3973 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.342309 3297 leader_election.cc:288] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 13 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:20.343820 3973 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:20.346226 31554 leader_election.cc:334] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 11 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.324s (SENT)
I20260805 22:25:20.347085 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:20.347193 3788 raft_consensus.cc:504] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:20.347884 31711 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
W20260805 22:25:20.350394 31554 leader_election.cc:334] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.332s (SENT)
I20260805 22:25:20.351987 31554 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:20.349253 3788 leader_election.cc:288] T b185bcb30c384758983895f8e5d8c98a P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 12 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:20.352981 3776 raft_consensus.cc:2781] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260805 22:25:20.353372 3776 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:20.353739 3776 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:20.355895 31444 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:20.353411 32586 leader_election.cc:288] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:20.357434 3328 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:20.357851 3328 raft_consensus.cc:482] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:20.358147 3328 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:20.359758 3786 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:20.360289 3786 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.361776 3818 consensus_queue.cc:1035] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:25:20.362320 3786 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:20.362951 3896 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:20.363420 3896 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:20.364729 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:20.365343 31705 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 8.
I20260805 22:25:20.365288 3896 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
W20260805 22:25:20.366178 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 63156.6us
W20260805 22:25:20.366844 32529 leader_election.cc:334] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.775s (SENT)
I20260805 22:25:20.368947 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
W20260805 22:25:20.337090 31667 leader_election.cc:334] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.766s (SENT)
I20260805 22:25:20.374468 3328 raft_consensus.cc:504] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.375214 3776 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.375478 3771 consensus_queue.cc:1035] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260805 22:25:20.375509 31554 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:20.379951 31667 leader_election.cc:302] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:20.380168 31502 raft_consensus.cc:1261] T 07310455e35e48f8ba4eb9b72a69113b P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 5 index: 3. Preceding OpId from leader: term: 5 index: 4. (index mismatch)
I20260805 22:25:20.386991 31608 raft_consensus.cc:1226] T 2d7039038a134310977029d3b3fb9fc9 P 516cb07ce3064ccbbca35e912d62816c [term 6 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:20.385914 32529 leader_election.cc:302] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:20.388518 32529 leader_election.cc:302] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, 6bf8c694a0ba40978195ae044a917caf; no voters:
I20260805 22:25:20.330708 3197 raft_consensus.cc:2726] T 6b16a2e0814140e39609da87479bb57e P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:25:20.397294 31609 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:20.395756 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:20.400382 31502 raft_consensus.cc:1261] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 9 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260805 22:25:20.401088 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:20.398579 3986 raft_consensus.cc:482] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (detected failure of leader 6bf8c694a0ba40978195ae044a917caf)
I20260805 22:25:20.402458 3986 raft_consensus.cc:504] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
W20260805 22:25:20.404387 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 67363.3us
I20260805 22:25:20.396914 3788 raft_consensus.cc:2781] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:20.405185 3788 raft_consensus.cc:482] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:20.377345 3776 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:20.408281 31609 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 10.
I20260805 22:25:20.401765 31709 raft_consensus.cc:3037] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Advancing to term 7
I20260805 22:25:20.377456 3328 leader_election.cc:288] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:20.409315 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:20.416036 3906 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:20.416812 3605 raft_consensus.cc:2781] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader election won for term 8
I20260805 22:25:20.417340 3605 raft_consensus.cc:686] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [term 8 LEADER]: Becoming Leader. State: Replica: 6bf8c694a0ba40978195ae044a917caf, State: Running, Role: LEADER
I20260805 22:25:20.418258 3605 consensus_queue.cc:237] T 7c99e6a491454b5ba8d31e3246e77797 P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 6.7, Last appended by leader: 7, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.394685 3771 raft_consensus.cc:2726] T f0eb14e6bb5c4855859507799f2b9e16 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:20.423223 3896 consensus_queue.cc:1035] T b185bcb30c384758983895f8e5d8c98a P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260805 22:25:20.404976 3765 raft_consensus.cc:2726] T 45cd56d4e96040f98dbf1ae001a1005a P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:20.394516 3857 consensus_queue.cc:1035] T ab66683a9e5e46c2b09be4afdee2a08c P 6bf8c694a0ba40978195ae044a917caf [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.036s
I20260805 22:25:20.429504 29783 auto_leader_rebalancer.cc:140] leader rebalance for table test-workload
I20260805 22:25:20.429529 31667 leader_election.cc:302] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:20.430555 3910 raft_consensus.cc:2781] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader election won for term 10
I20260805 22:25:20.431034 3910 raft_consensus.cc:686] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:20.431878 3910 consensus_queue.cc:237] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 8.9, Last appended by leader: 9, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.433499 31501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.433871 31500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab66683a9e5e46c2b09be4afdee2a08c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.434259 31501 raft_consensus.cc:2379] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 7: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:20.435029 31492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.435681 31492 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:25:20.436273 31486 raft_consensus.cc:1226] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 13 FOLLOWER]: Rejecting Update request from peer 516cb07ce3064ccbbca35e912d62816c for earlier term 12. Current term is 13. Ops: []
I20260805 22:25:20.436410 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:20.435118 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:20.437295 31495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.437582 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.438126 31495 raft_consensus.cc:2379] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 11: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:20.438337 31493 raft_consensus.cc:2379] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 9: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:20.405521 3788 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:20.440137 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eda619941ce64f788d0b7a364eed0bdb" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.440852 31502 raft_consensus.cc:2379] T eda619941ce64f788d0b7a364eed0bdb P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:20.437728 31499 raft_consensus.cc:2396] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 10 because replica has last-logged OpId of term: 9 index: 9, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 8.
I20260805 22:25:20.443980 3771 raft_consensus.cc:482] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:20.456677 31609 raft_consensus.cc:1226] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
I20260805 22:25:20.404690 3986 leader_election.cc:288] T e6cf90ae3fc840ceb48be8c790be97d9 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:20.405336 3844 raft_consensus.cc:482] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader f8e9f4c756da4203917c464e2b179994)
I20260805 22:25:20.466312 31444 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:20.468001 3328 consensus_queue.cc:1046] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 5.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260805 22:25:20.469555 3328 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260805 22:25:20.469970 3328 raft_consensus.cc:482] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:20.470283 3328 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Advancing to term 11
W20260805 22:25:20.472533 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.194:41897, user_credentials={real_user=slave}} blocked reactor thread for 84057.7us
I20260805 22:25:20.473443 32529 leader_election.cc:302] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:20.478327 31708 raft_consensus.cc:2454] T 17b3240dd1b947178dafdc9cac7f1e5e P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 11.
I20260805 22:25:20.478456 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89a7aa19faed455fb2600ae6851aee0e" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 6 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:20.479211 31597 raft_consensus.cc:2454] T 89a7aa19faed455fb2600ae6851aee0e P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 7.
I20260805 22:25:20.480050 31608 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:20.480687 31608 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:20.483870 31596 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:20.485157 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.486744 31554 leader_election.cc:302] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:20.464653 3896 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:20.489711 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 6 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.471657 3297 consensus_queue.cc:1046] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.013s
I20260805 22:25:20.468837 3788 raft_consensus.cc:504] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.498843 31486 raft_consensus.cc:2379] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 6: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:20.500041 31494 raft_consensus.cc:2379] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 10: Already voted for candidate f8e9f4c756da4203917c464e2b179994 in this term.
I20260805 22:25:20.491173 3197 raft_consensus.cc:3032] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Stepping down as leader of term 5
I20260805 22:25:20.491039 3986 raft_consensus.cc:482] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:20.506841 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39e7b32cb2f646828a02f900ec2d47f1" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 10 candidate_status { last_received { term: 8 index: 9 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:20.507717 3996 raft_consensus.cc:2726] T 3f76fdd524184682ae27abc293a7fff6 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:20.510635 3788 leader_election.cc:288] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 election: Requested vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:20.512224 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:20.515012 32586 raft_consensus.cc:3037] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 9 FOLLOWER]: Advancing to term 10
I20260805 22:25:20.462986 3771 raft_consensus.cc:504] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.479353 3844 raft_consensus.cc:504] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:20.522753 31596 raft_consensus.cc:2379] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 9: Already voted for candidate 516cb07ce3064ccbbca35e912d62816c in this term.
I20260805 22:25:20.529171 32581 raft_consensus.cc:2379] T aee974c178e84c3d9a5dcfb31556a46f P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader election vote request: Denying vote to candidate feb5e7ab4b1541beaf2674c3c5500385 in current term 5: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:20.499672 3776 raft_consensus.cc:2781] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Leader election won for term 11
I20260805 22:25:20.518754 31709 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 7.
I20260805 22:25:20.536523 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.545826 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:20.528889 3328 raft_consensus.cc:504] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.546589 32584 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 8.
W20260805 22:25:20.539400 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 45545.8us
I20260805 22:25:20.548942 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:20.549593 31709 raft_consensus.cc:3037] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:20.549840 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:20.550448 31708 raft_consensus.cc:2454] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 5.
I20260805 22:25:20.514252 3197 raft_consensus.cc:729] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:20.550137 32586 raft_consensus.cc:2454] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 10.
W20260805 22:25:20.565850 31554 leader_election.cc:334] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 12 election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.987s (SENT)
I20260805 22:25:20.567178 31554 leader_election.cc:302] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 7 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:20.527994 3986 raft_consensus.cc:504] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:20.569015 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aee974c178e84c3d9a5dcfb31556a46f" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 6 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:20.501161 3896 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:20.571941 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.572718 31494 raft_consensus.cc:2354] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 8. Current term is 9.
I20260805 22:25:20.576488 3844 leader_election.cc:288] T 527d60ab6db74595bed7e87ca2ecc6e8 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:20.581530 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.582335 31502 raft_consensus.cc:2354] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 10. Current term is 11.
I20260805 22:25:20.582341 31486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10de517cc0a24f20b5885091e99e252d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 6 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.601596 31668 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:20.584892 31709 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 9.
W20260805 22:25:20.603003 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 123488us
I20260805 22:25:20.620221 31554 leader_election.cc:302] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:20.622912 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: true dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:20.626408 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 12 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:20.627209 31502 raft_consensus.cc:2354] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 13 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for earlier term 12. Current term is 13.
I20260805 22:25:20.629715 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f0eb14e6bb5c4855859507799f2b9e16" candidate_uuid: "6bf8c694a0ba40978195ae044a917caf" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: true dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:20.630401 31708 raft_consensus.cc:3032] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 LEADER]: Stepping down as leader of term 7
I20260805 22:25:20.630694 31708 raft_consensus.cc:729] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:20.552637 3776 raft_consensus.cc:686] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 11 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:20.632715 31499 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e4ad7072aab4331878e988a26a6d887" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:20.633083 31709 raft_consensus.cc:1261] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Refusing update from remote peer f8e9f4c756da4203917c464e2b179994: Log matching property violated. Preceding OpId in replica: term: 6 index: 5. Preceding OpId from leader: term: 7 index: 6. (index mismatch)
I20260805 22:25:20.647112 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.632272 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.649427 31494 raft_consensus.cc:2362] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 10. Re-sending same reply.
I20260805 22:25:20.649632 31490 raft_consensus.cc:2396] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
W20260805 22:25:20.646811 31668 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.196:41801, user_credentials={real_user=slave}} blocked reactor thread for 45422.1us
I20260805 22:25:20.652196 31603 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:20.652966 31597 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:20.646281 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "18a7e330200a49349edcae379f1e2b57" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.655228 31493 raft_consensus.cc:2379] T 18a7e330200a49349edcae379f1e2b57 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 9: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:20.656606 2956 raft_consensus.cc:2741] T bd1c7254b2b74c10a4fb473ea633544d P feb5e7ab4b1541beaf2674c3c5500385 [term 9 FOLLOWER]: Leader pre-election decision vote started in defunct term 8: won
I20260805 22:25:20.660238 3788 raft_consensus.cc:2781] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election won for term 9
I20260805 22:25:20.660426 3771 leader_election.cc:288] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:20.660863 3788 raft_consensus.cc:686] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [term 9 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:20.601048 3873 raft_consensus.cc:2781] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 7 FOLLOWER]: Leader election won for term 7
I20260805 22:25:20.661473 3873 raft_consensus.cc:686] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [term 7 LEADER]: Becoming Leader. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:20.661782 3788 consensus_queue.cc:237] T bd1c7254b2b74c10a4fb473ea633544d P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 7.7, Last appended by leader: 7, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.661414 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4deaa72a885a43bfbc6171a7ed13ad81" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 13 candidate_status { last_received { term: 11 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:20.668684 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:20.671767 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f8863c96c162438983f403d38b11c5db" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:20.658772 31708 consensus_queue.cc:260] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 7, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.650417 31594 raft_consensus.cc:2454] T aee974c178e84c3d9a5dcfb31556a46f P 516cb07ce3064ccbbca35e912d62816c [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 5.
I20260805 22:25:20.679351 31604 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "527d60ab6db74595bed7e87ca2ecc6e8" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 8 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:20.681990 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 114612us
I20260805 22:25:20.682683 31499 raft_consensus.cc:2454] T 5e4ad7072aab4331878e988a26a6d887 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 10.
I20260805 22:25:20.683491 31608 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 9.
I20260805 22:25:20.647877 3297 raft_consensus.cc:482] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
W20260805 22:25:20.691160 31667 leader_election.cc:334] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 12 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.922s (SENT)
I20260805 22:25:20.657652 3838 raft_consensus.cc:482] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:20.693784 31708 raft_consensus.cc:3037] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:20.695992 31667 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 12 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:20.676999 3907 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
W20260805 22:25:20.705128 32529 leader_election.cc:334] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 10 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.984s (SENT)
I20260805 22:25:20.705811 31597 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 8.
I20260805 22:25:20.685540 3986 leader_election.cc:288] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:20.714214 31708 raft_consensus.cc:2454] T f0eb14e6bb5c4855859507799f2b9e16 P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bf8c694a0ba40978195ae044a917caf in term 8.
I20260805 22:25:20.698722 3297 raft_consensus.cc:504] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.715587 3328 leader_election.cc:288] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 11 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:20.717371 3297 leader_election.cc:288] T 36ee93639c3f4084b94145c6ee60a2a8 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
I20260805 22:25:20.717957 3907 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:20.720217 3907 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:20.691884 3736 consensus_queue.cc:1035] T 89a7aa19faed455fb2600ae6851aee0e P f8e9f4c756da4203917c464e2b179994 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.027s
I20260805 22:25:20.698248 3910 raft_consensus.cc:2726] T 4deaa72a885a43bfbc6171a7ed13ad81 P feb5e7ab4b1541beaf2674c3c5500385 [term 12 FOLLOWER]: Leader pre-election lost for term 12. Reason: could not achieve majority
W20260805 22:25:20.730459 31444 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 51874.9us
I20260805 22:25:20.732657 31597 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:20.734280 31597 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:20.735229 31444 leader_election.cc:302] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:20.739369 31494 raft_consensus.cc:1261] T 986c28960d44411487f79ca134af59e2 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 6 index: 7. (index mismatch)
I20260805 22:25:20.742615 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:20.743667 31493 raft_consensus.cc:1261] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Refusing update from remote peer 516cb07ce3064ccbbca35e912d62816c: Log matching property violated. Preceding OpId in replica: term: 8 index: 5. Preceding OpId from leader: term: 10 index: 6. (index mismatch)
I20260805 22:25:20.659637 3197 consensus_queue.cc:260] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.749275 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.749928 31494 raft_consensus.cc:2454] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 8.
I20260805 22:25:20.708292 3776 consensus_queue.cc:237] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 8.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:20.710067 3838 raft_consensus.cc:504] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.755488 3838 leader_election.cc:288] T dc38ff59fd654f6689e3a88fdbb765cf P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831), 6bf8c694a0ba40978195ae044a917caf (127.29.21.196:41801)
W20260805 22:25:20.756405 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 73373.5us
W20260805 22:25:20.757151 31667 leader_election.cc:334] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.980s (SENT)
I20260805 22:25:20.757689 31667 leader_election.cc:302] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:20.758755 3910 raft_consensus.cc:2726] T 8abfbdc4afff482ebc81c4fda5d0812a P feb5e7ab4b1541beaf2674c3c5500385 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260805 22:25:20.760937 31597 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 11.
I20260805 22:25:20.762001 3800 raft_consensus.cc:3032] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 10 LEADER]: Stepping down as leader of term 10
I20260805 22:25:20.762316 31444 leader_election.cc:302] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:20.763773 3838 raft_consensus.cc:2781] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Leader election won for term 11
W20260805 22:25:20.765821 32529 leader_election.cc:334] T 8513af2d574d45b4a6d02428543b1402 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.020s (SENT)
I20260805 22:25:20.739701 3818 raft_consensus.cc:2781] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader election won for term 9
W20260805 22:25:20.767241 31667 leader_election.cc:334] T 32fee80a830a450195ce9c3462460ca1 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.661s (SENT)
I20260805 22:25:20.768359 3818 raft_consensus.cc:686] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [term 9 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:20.769121 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.769296 3818 consensus_queue.cc:237] T 78cb483d17014b3281316012dbb6f6e1 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 7.8, Last appended by leader: 8, Current term: 9, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.769930 3197 raft_consensus.cc:3037] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 5 FOLLOWER]: Advancing to term 6
W20260805 22:25:20.767223 31554 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.21.195:44831, user_credentials={real_user=slave}} blocked reactor thread for 90734us
I20260805 22:25:20.764271 3838 raft_consensus.cc:686] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [term 11 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:20.805539 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a260d4eadd0a4677aed088f331367c05" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 7 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:20.762398 3800 raft_consensus.cc:729] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 10 LEADER]: Becoming Follower/Learner. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:20.806552 31494 raft_consensus.cc:2396] T a260d4eadd0a4677aed088f331367c05 P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 11 because replica has last-logged OpId of term: 10 index: 7, which is greater than that of the candidate, which has last-logged OpId of term: 7 index: 5.
I20260805 22:25:20.807296 3896 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:20.852470 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe370453251f4300a8d90c397021595c" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:20.863308 31602 raft_consensus.cc:1226] T 17b3240dd1b947178dafdc9cac7f1e5e P 516cb07ce3064ccbbca35e912d62816c [term 11 LEADER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 10. Current term is 11. Ops: [10.6-10.6]
W20260805 22:25:20.877709 31667 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 110548us
W20260805 22:25:20.878480 31667 leader_election.cc:334] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.126s (SENT)
I20260805 22:25:20.878880 31667 leader_election.cc:302] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
W20260805 22:25:20.880097 31667 leader_election.cc:334] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.137s (SENT)
I20260805 22:25:20.880499 31667 leader_election.cc:302] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:20.881789 3878 raft_consensus.cc:2726] T 6a1aa2dc5e5341f0a0450361d6c95814 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:20.882380 3878 raft_consensus.cc:2726] T 52af4d94f396434d997b46f52763cd20 P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260805 22:25:20.885921 3838 consensus_queue.cc:237] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 9.9, Last appended by leader: 9, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.887194 3328 raft_consensus.cc:482] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:20.887696 3328 raft_consensus.cc:504] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:20.747640 3873 consensus_queue.cc:237] T 42324773ce144eb784e14693eecbee72 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 5.6, Last appended by leader: 6, Current term: 7, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.889706 3328 leader_election.cc:288] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:20.891237 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
W20260805 22:25:20.864287 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 98515.3us
I20260805 22:25:20.808588 3907 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:20.897698 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:20.895813 3907 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:20.901046 3907 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
W20260805 22:25:20.903137 32529 leader_election.cc:334] T 1115c6a30f64413ab24c9d215d58a2c7 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 8 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.122s (SENT)
I20260805 22:25:20.905014 31602 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:20.853456 3904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07310455e35e48f8ba4eb9b72a69113b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260805 22:25:20.912477 31602 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 9.
W20260805 22:25:20.910931 31667 leader_election.cc:334] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.153s (SENT)
I20260805 22:25:20.915570 31444 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:20.919533 31667 leader_election.cc:302] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:20.922140 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd72177fb2f24347ba19da011ccd3fe4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 9 index: 4 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:20.923058 31493 raft_consensus.cc:2379] T cd72177fb2f24347ba19da011ccd3fe4 P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 11: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:20.922763 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ff925b5d86b41c19d809f76e57ab062" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:20.925513 31502 raft_consensus.cc:2362] T 0ff925b5d86b41c19d809f76e57ab062 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader election vote request: Already granted yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 10. Re-sending same reply.
I20260805 22:25:20.934046 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3095cdb25afc4c26bd9d1e03a919b474" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:20.921576 3328 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260805 22:25:20.939288 31493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17b3240dd1b947178dafdc9cac7f1e5e" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 11 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
I20260805 22:25:20.941505 31502 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8513af2d574d45b4a6d02428543b1402" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 8 index: 10 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.927143 3878 raft_consensus.cc:2726] T 6b16a2e0814140e39609da87479bb57e P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:25:20.943570 31667 leader_election.cc:302] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
I20260805 22:25:20.944096 31502 raft_consensus.cc:2379] T 8513af2d574d45b4a6d02428543b1402 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c in current term 9: Already voted for candidate 6bf8c694a0ba40978195ae044a917caf in this term.
I20260805 22:25:20.944948 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 10 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994"
W20260805 22:25:20.947314 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 44262.4us
I20260805 22:25:20.940635 3328 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:20.951120 31490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b185bcb30c384758983895f8e5d8c98a" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 12 candidate_status { last_received { term: 9 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.951798 31490 raft_consensus.cc:2454] T b185bcb30c384758983895f8e5d8c98a P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 11.
I20260805 22:25:20.955117 3328 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:20.957222 31667 leader_election.cc:302] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf
I20260805 22:25:20.958403 3505 raft_consensus.cc:2781] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 5 FOLLOWER]: Leader election won for term 5
I20260805 22:25:20.954684 32581 raft_consensus.cc:1226] T 58a08ace2d684c14972d495f4a387a85 P 6bf8c694a0ba40978195ae044a917caf [term 7 LEADER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 6. Current term is 7. Ops: [6.6-6.6]
I20260805 22:25:20.958902 3505 raft_consensus.cc:686] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [term 5 LEADER]: Becoming Leader. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:20.959830 3505 consensus_queue.cc:237] T aee974c178e84c3d9a5dcfb31556a46f P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 4.6, Last appended by leader: 6, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } } peers { permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false } }
I20260805 22:25:20.963858 31494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "f8e9f4c756da4203917c464e2b179994" is_pre_election: true
I20260805 22:25:20.964321 32581 raft_consensus.cc:1226] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 5. Current term is 6. Ops: []
I20260805 22:25:20.964551 31494 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:25:20.967581 32589 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P 6bf8c694a0ba40978195ae044a917caf [term 10 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260805 22:25:20.971596 31444 leader_election.cc:302] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 13 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
W20260805 22:25:20.975085 32529 leader_election.cc:334] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.125s (SENT)
I20260805 22:25:20.975591 32529 leader_election.cc:302] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6bf8c694a0ba40978195ae044a917caf; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:20.978226 31444 leader_election.cc:302] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:20.977135 3984 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bf8c694a0ba40978195ae044a917caf" member_type: VOTER last_known_addr { host: "127.29.21.196" port: 41801 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.003s
W20260805 22:25:20.986990 31667 leader_election.cc:334] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 1.833s (SENT)
I20260805 22:25:20.987463 31667 leader_election.cc:302] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: feb5e7ab4b1541beaf2674c3c5500385; no voters: 6bf8c694a0ba40978195ae044a917caf, f8e9f4c756da4203917c464e2b179994
I20260805 22:25:20.989785 2956 raft_consensus.cc:2726] T d8fd526bfd0047e49803dbf11f8eafc6 P feb5e7ab4b1541beaf2674c3c5500385 [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
I20260805 22:25:20.990278 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "feb5e7ab4b1541beaf2674c3c5500385" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:20.990962 31600 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate feb5e7ab4b1541beaf2674c3c5500385 in term 9.
I20260805 22:25:20.995337 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:20.996471 31709 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8dc633587f48d1bba2a54a1619bc3e" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 7 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:20.937516 3800 consensus_queue.cc:260] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 10.6, Last appended by leader: 6, Current term: 10, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:20.994355 3986 raft_consensus.cc:2726] T f8863c96c162438983f403d38b11c5db P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Leader pre-election lost for term 8. Reason: could not achieve majority
I20260805 22:25:20.997944 3605 raft_consensus.cc:2726] T 9d68a856424b4ceb8f4f54b4050e8dc4 P 6bf8c694a0ba40978195ae044a917caf [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260805 22:25:20.998901 32581 raft_consensus.cc:1226] T fe370453251f4300a8d90c397021595c P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 7. Current term is 8. Ops: []
I20260805 22:25:21.021883 4008 consensus_queue.cc:1046] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Peer responded invalid term: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 4, Time since last communication: 0.001s
I20260805 22:25:21.022706 4008 consensus_queue.cc:1225] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [NON_LEADER]: Peer 516cb07ce3064ccbbca35e912d62816c log is divergent from this leader: its last log entry 11.6 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20260805 22:25:20.999118 31705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:21.026064 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c75e772a0f924080865ddf85ed1e2d86" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:21.026870 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:21.027549 32585 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 8.
I20260805 22:25:21.000913 31708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f41c2ff98f049c3bc654aa98cd77694" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 10 candidate_status { last_received { term: 8 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:20.999567 3328 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:21.004052 31704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5df9cd03cde746e0956c8c7726d4eb70" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 4 index: 4 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:21.004683 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f914c8243e724eda849ce1fc2dc30827" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:21.005311 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385" is_pre_election: true
I20260805 22:25:21.006196 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a9acdbc98a4e799374fe6e8950c3a7" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 8 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:21.031445 31706 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 10.
I20260805 22:25:21.010414 31600 raft_consensus.cc:1261] T 39e7b32cb2f646828a02f900ec2d47f1 P 516cb07ce3064ccbbca35e912d62816c [term 10 FOLLOWER]: Refusing update from remote peer feb5e7ab4b1541beaf2674c3c5500385: Log matching property violated. Preceding OpId in replica: term: 8 index: 9. Preceding OpId from leader: term: 10 index: 10. (index mismatch)
I20260805 22:25:21.009476 3769 raft_consensus.cc:2781] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:21.032896 3769 raft_consensus.cc:482] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:21.031739 31444 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:21.033351 3769 raft_consensus.cc:3037] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:21.034017 3800 raft_consensus.cc:3037] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:21.034842 3986 raft_consensus.cc:2726] T fb8dc633587f48d1bba2a54a1619bc3e P f8e9f4c756da4203917c464e2b179994 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260805 22:25:21.035432 3328 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 11 election: Requested vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:21.035097 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c"
I20260805 22:25:21.036715 31600 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Advancing to term 11
I20260805 22:25:20.994686 3818 raft_consensus.cc:2726] T 4deaa72a885a43bfbc6171a7ed13ad81 P f8e9f4c756da4203917c464e2b179994 [term 13 FOLLOWER]: Leader election lost for term 13. Reason: could not achieve majority
I20260805 22:25:21.045848 31667 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385; no voters:
W20260805 22:25:21.047607 32529 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.21.193:45607, user_credentials={real_user=slave}} blocked reactor thread for 75167.2us
I20260805 22:25:21.048925 3984 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Leader pre-election won for term 11
I20260805 22:25:21.049413 3984 raft_consensus.cc:482] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260805 22:25:21.049760 3984 raft_consensus.cc:3037] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:21.054080 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:21.054816 32584 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 10 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 10.
I20260805 22:25:21.063194 31493 raft_consensus.cc:2396] T 17b3240dd1b947178dafdc9cac7f1e5e P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate 516cb07ce3064ccbbca35e912d62816c for term 11 because replica has last-logged OpId of term: 10 index: 6, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 5.
I20260805 22:25:21.064628 3984 raft_consensus.cc:504] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
W20260805 22:25:21.065770 32529 leader_election.cc:334] T 2d7039038a134310977029d3b3fb9fc9 P 6bf8c694a0ba40978195ae044a917caf [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607): Timed out: RequestConsensusVote RPC to 127.29.21.193:45607 timed out after 2.157s (SENT)
I20260805 22:25:21.068393 31600 raft_consensus.cc:2454] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P 516cb07ce3064ccbbca35e912d62816c [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 11.
I20260805 22:25:21.068578 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "78cb483d17014b3281316012dbb6f6e1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 9 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:21.069226 32586 raft_consensus.cc:3037] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:21.069502 31444 leader_election.cc:302] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 11 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters:
I20260805 22:25:21.070353 3800 raft_consensus.cc:2781] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 11 FOLLOWER]: Leader election won for term 11
I20260805 22:25:21.070894 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcea14dac2934d139e2c5ab6d0b8648d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 6 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:21.071012 3800 raft_consensus.cc:686] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [term 11 LEADER]: Becoming Leader. State: Replica: f8e9f4c756da4203917c464e2b179994, State: Running, Role: LEADER
I20260805 22:25:21.072113 3800 consensus_queue.cc:237] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P f8e9f4c756da4203917c464e2b179994 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 9.5, Last appended by leader: 5, Current term: 11, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:21.075138 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6cf90ae3fc840ceb48be8c790be97d9" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 7 candidate_status { last_received { term: 6 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:21.081241 31444 leader_election.cc:302] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:21.082974 31444 leader_election.cc:302] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f8e9f4c756da4203917c464e2b179994; no voters: 516cb07ce3064ccbbca35e912d62816c, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:21.086585 32586 raft_consensus.cc:2454] T 78cb483d17014b3281316012dbb6f6e1 P 6bf8c694a0ba40978195ae044a917caf [term 9 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 9.
I20260805 22:25:21.089790 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36ee93639c3f4084b94145c6ee60a2a8" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 7 index: 8 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:21.087649 3818 raft_consensus.cc:2726] T 9f41c2ff98f049c3bc654aa98cd77694 P f8e9f4c756da4203917c464e2b179994 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260805 22:25:21.093935 3197 raft_consensus.cc:2726] T 5df9cd03cde746e0956c8c7726d4eb70 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Leader pre-election lost for term 7. Reason: could not achieve majority
I20260805 22:25:21.094612 3328 raft_consensus.cc:482] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260805 22:25:21.095144 3328 raft_consensus.cc:504] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:21.096064 32581 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c43b2a80b8c446bdb2ac0e9ca04e8cf4" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 11 candidate_status { last_received { term: 9 index: 9 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:21.096635 32581 raft_consensus.cc:3037] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 10 FOLLOWER]: Advancing to term 11
I20260805 22:25:21.093746 3873 consensus_queue.cc:1046] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 8.5, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:25:21.060900 3769 raft_consensus.cc:504] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:21.098814 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dc38ff59fd654f6689e3a88fdbb765cf" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 5 candidate_status { last_received { term: 4 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:21.101770 31706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6f372cf6a3ec4a4cac2b6335a1a64a56" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 11 candidate_status { last_received { term: 9 index: 5 } } ignore_live_leader: false dest_uuid: "feb5e7ab4b1541beaf2674c3c5500385"
I20260805 22:25:21.102497 31706 raft_consensus.cc:2379] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [term 11 FOLLOWER]: Leader election vote request: Denying vote to candidate f8e9f4c756da4203917c464e2b179994 in current term 11: Already voted for candidate feb5e7ab4b1541beaf2674c3c5500385 in this term.
I20260805 22:25:21.104439 31600 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32fee80a830a450195ce9c3462460ca1" candidate_uuid: "f8e9f4c756da4203917c464e2b179994" candidate_term: 8 candidate_status { last_received { term: 5 index: 3 } } ignore_live_leader: false dest_uuid: "516cb07ce3064ccbbca35e912d62816c" is_pre_election: true
I20260805 22:25:21.104787 32584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d264e8379ab54b4785c836a00397ba95" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:21.105767 4005 consensus_queue.cc:1035] T 5e4ad7072aab4331878e988a26a6d887 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260805 22:25:21.104274 3786 consensus_queue.cc:1035] T 07310455e35e48f8ba4eb9b72a69113b P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20260805 22:25:21.108155 3873 raft_consensus.cc:3032] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 8 LEADER]: Stepping down as leader of term 8
I20260805 22:25:21.108523 3873 raft_consensus.cc:729] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:21.109396 3873 consensus_queue.cc:260] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 8.5, Last appended by leader: 5, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:21.104467 3973 consensus_queue.cc:1046] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.009s
I20260805 22:25:21.113700 3910 consensus_queue.cc:1046] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: INVALID_TERM, Last received: 4.6, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260805 22:25:21.082553 3896 consensus_queue.cc:1035] T 39e7b32cb2f646828a02f900ec2d47f1 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.032s
I20260805 22:25:21.114161 31493 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0191f03fa8984a40afc7ec1e44970c8e"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:25:21.114995 31493 raft_consensus.cc:595] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:21.115439 31493 raft_consensus.cc:603] T 0191f03fa8984a40afc7ec1e44970c8e P f8e9f4c756da4203917c464e2b179994 [term 4 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:25:21.117269 31711 raft_consensus.cc:1226] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [term 8 FOLLOWER]: Rejecting Update request from peer f8e9f4c756da4203917c464e2b179994 for earlier term 7. Current term is 8. Ops: [7.6-7.6]
I20260805 22:25:21.097266 3328 leader_election.cc:288] T 32fee80a830a450195ce9c3462460ca1 P f8e9f4c756da4203917c464e2b179994 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:21.117856 32584 raft_consensus.cc:2454] T d264e8379ab54b4785c836a00397ba95 P 6bf8c694a0ba40978195ae044a917caf [term 6 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 6.
I20260805 22:25:21.118860 32581 raft_consensus.cc:2454] T c43b2a80b8c446bdb2ac0e9ca04e8cf4 P 6bf8c694a0ba40978195ae044a917caf [term 11 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8e9f4c756da4203917c464e2b179994 in term 11.
I20260805 22:25:21.114923 3910 raft_consensus.cc:3032] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Stepping down as leader of term 4
I20260805 22:25:21.100625 3984 leader_election.cc:288] T 6f372cf6a3ec4a4cac2b6335a1a64a56 P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 11 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:21.119792 3910 raft_consensus.cc:729] T 9d68a856424b4ceb8f4f54b4050e8dc4 P feb5e7ab4b1541beaf2674c3c5500385 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: feb5e7ab4b1541beaf2674c3c5500385, State: Running, Role: LEADER
I20260805 22:25:21.122608 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:21.123152 32586 raft_consensus.cc:2454] T 42324773ce144eb784e14693eecbee72 P 6bf8c694a0ba40978195ae044a917caf [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 5.
I20260805 22:25:21.110785 3873 raft_consensus.cc:3037] T 9f41c2ff98f049c3bc654aa98cd77694 P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:21.125689 31554 leader_election.cc:302] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c, f8e9f4c756da4203917c464e2b179994; no voters: feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:21.125903 4000 consensus_queue.cc:1035] T 0ff925b5d86b41c19d809f76e57ab062 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260805 22:25:21.126525 3786 raft_consensus.cc:2781] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260805 22:25:21.126875 3786 raft_consensus.cc:482] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Starting leader election (detected failure of leader feb5e7ab4b1541beaf2674c3c5500385)
I20260805 22:25:21.127223 3786 raft_consensus.cc:3037] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 8 FOLLOWER]: Advancing to term 9
I20260805 22:25:21.132071 32586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf" is_pre_election: true
I20260805 22:25:21.132807 32586 raft_consensus.cc:2454] T bd1c7254b2b74c10a4fb473ea633544d P 6bf8c694a0ba40978195ae044a917caf [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 516cb07ce3064ccbbca35e912d62816c in term 8.
I20260805 22:25:21.131494 31554 leader_election.cc:302] T a260d4eadd0a4677aed088f331367c05 P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 11 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 516cb07ce3064ccbbca35e912d62816c; no voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385
I20260805 22:25:21.130659 3973 consensus_queue.cc:1035] T 986c28960d44411487f79ca134af59e2 P 516cb07ce3064ccbbca35e912d62816c [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.006s
I20260805 22:25:21.136570 31493 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "2d7039038a134310977029d3b3fb9fc9"
dest_uuid: "f8e9f4c756da4203917c464e2b179994"
mode: GRACEFUL
new_leader_uuid: "516cb07ce3064ccbbca35e912d62816c"
from {username='slave'} at 127.0.0.1:54616
I20260805 22:25:21.137814 31493 raft_consensus.cc:595] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Received request to transfer leadership to TS 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:21.138250 31493 raft_consensus.cc:603] T 2d7039038a134310977029d3b3fb9fc9 P f8e9f4c756da4203917c464e2b179994 [term 6 FOLLOWER]: Rejecting request to transer leadership while not leader
I20260805 22:25:21.139458 3505 consensus_queue.cc:1035] T 18a7e330200a49349edcae379f1e2b57 P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:25:21.140017 31667 leader_election.cc:302] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: f8e9f4c756da4203917c464e2b179994, feb5e7ab4b1541beaf2674c3c5500385; no voters: 516cb07ce3064ccbbca35e912d62816c
I20260805 22:25:21.140971 2956 consensus_queue.cc:1035] T eda619941ce64f788d0b7a364eed0bdb P feb5e7ab4b1541beaf2674c3c5500385 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20260805 22:25:21.141595 3878 raft_consensus.cc:2781] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260805 22:25:21.142037 3878 raft_consensus.cc:482] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Starting leader election (detected failure of leader 516cb07ce3064ccbbca35e912d62816c)
I20260805 22:25:21.142385 3878 raft_consensus.cc:3037] T c5b0fd7f6d7a46b2a77e841ceda358ef P feb5e7ab4b1541beaf2674c3c5500385 [term 7 FOLLOWER]: Advancing to term 8
I20260805 22:25:21.143328 3786 raft_consensus.cc:504] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:21.136570 3769 leader_election.cc:288] T 89a7aa19faed455fb2600ae6851aee0e P feb5e7ab4b1541beaf2674c3c5500385 [CANDIDATE]: Term 8 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), 516cb07ce3064ccbbca35e912d62816c (127.29.21.194:41897)
I20260805 22:25:21.147588 3786 leader_election.cc:288] T fb8dc633587f48d1bba2a54a1619bc3e P 516cb07ce3064ccbbca35e912d62816c [CANDIDATE]: Term 9 election: Requested vote from peers f8e9f4c756da4203917c464e2b179994 (127.29.21.193:45607), feb5e7ab4b1541beaf2674c3c5500385 (127.29.21.195:44831)
I20260805 22:25:21.149065 4013 raft_consensus.cc:3032] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 12 LEADER]: Stepping down as leader of term 12
I20260805 22:25:21.153585 4013 raft_consensus.cc:729] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 12 LEADER]: Becoming Follower/Learner. State: Replica: 516cb07ce3064ccbbca35e912d62816c, State: Running, Role: LEADER
I20260805 22:25:21.154673 4013 consensus_queue.cc:260] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 12.8, Last appended by leader: 8, Current term: 12, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f8e9f4c756da4203917c464e2b179994" member_type: VOTER last_known_addr { host: "127.29.21.193" port: 45607 } } peers { permanent_uuid: "516cb07ce3064ccbbca35e912d62816c" member_type: VOTER last_known_addr { host: "127.29.21.194" port: 41897 } } peers { permanent_uuid: "feb5e7ab4b1541beaf2674c3c5500385" member_type: VOTER last_known_addr { host: "127.29.21.195" port: 44831 } }
I20260805 22:25:21.156188 4013 raft_consensus.cc:3037] T 4deaa72a885a43bfbc6171a7ed13ad81 P 516cb07ce3064ccbbca35e912d62816c [term 12 FOLLOWER]: Advancing to term 13
I20260805 22:25:21.159492 32589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42324773ce144eb784e14693eecbee72" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 7 candidate_status { last_received { term: 5 index: 6 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"
I20260805 22:25:21.159761 32585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1c7254b2b74c10a4fb473ea633544d" candidate_uuid: "516cb07ce3064ccbbca35e912d62816c" candidate_term: 9 candidate_status { last_received { term: 7 index: 7 } } ignore_live_leader: false dest_uuid: "6bf8c694a0ba40978195ae044a917caf"